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Keynote:  The  Honorable  Sean  Stackley,  Assistant 
Secretary  of  the  Navy  for  Research,  Development, 
&  Acquisition 


The  Honorable  Sean  J.  Stackley — assumed  the  duties  of  assistant  secretary  of  the  Navy  (ASN), 
Research,  Development,  &  Acquisition  (RDA)  following  his  confirmation  by  the  Senate  in  July  2008. 
As  the  Navy’s  acquisition  executive,  Stackley  is  responsible  for  the  research,  development,  and 
acquisition  of  Navy  and  Marine  Corps  platforms  and  warfare  systems,  which  include  oversight  of 
more  than  100,000  people  and  an  annual  budget  in  excess  of  $50  billion. 

Prior  to  his  appointment  to  ASN  (RDA),  Stackley  served  as  a  professional  staff  member  of  the  Senate 
Armed  Services  Committee.  During  his  tenure  with  the  Committee,  he  was  responsible  for  overseeing 
Navy  and  Marine  Corps  programs,  U.S.  Transportation  Command  matters,  and  related  policy  for  the 
Seapower  Subcommittee.  He  also  advised  on  Navy  and  Marine  Corps  operations  and  maintenance, 
science  and  technology,  and  acquisition  policy. 

Stackley  began  his  career  as  a  Navy  surface  warfare  officer,  serving  in  engineering  and  combat 
systems  assignments  aboard  USS  John  Young  (DD  973).  Upon  completing  his  warfare  qualifications, 
he  was  designated  as  an  engineering  duty  officer  and  served  in  a  series  of  industrial,  fleet,  program 
office,  and  headquarters  assignments  in  ship  design  and  construction,  maintenance,  logistics,  and 
acquisition  policy. 

From  2001  to  2005,  Stackley  served  as  the  Navy’s  LPD  17  program  manager,  with  responsibility  for 
all  aspects  of  procurement  for  this  major  ship  program.  Having  served  earlier  in  his  career  as 
production  officer  for  the  USS  Arleigh  Burke  (DDG  51 )  and  project  naval  architect  overseeing 
structural  design  for  the  Canadian  Patrol  Frigate  HMCS  Halifax  (FFH  330),  he  has  the  unique 
experience  of  having  performed  a  principal  role  in  the  design,  construction,  test,  and  delivery  of  three 
first-of-class  warships. 

Stackley  was  commissioned  and  graduated  with  distinction  from  the  U.S.  Naval  Academy  in  1979 
with  a  Bachelor  of  Science  degree  in  mechanical  engineering.  He  holds  the  degrees  of  Ocean 
Engineer  and  Master  of  Science  in  mechanical  engineering  from  the  Massachusetts  Institute  of 
Technology.  Stackley  earned  certification  as  a  Commonwealth  of  Virginia  professional  engineer  in 
1994. 
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Plenary  Panel:  Weapon  Acquisition  Program 
Outcomes  and  Efforts  to  Reform  DoD’s  Acquisition 
Process 


Wednesday,  May  4,  2016 

9:30  a.m.  - 

Chair:  Michael  Sullivan,  Director,  GAO  Acquisition  and  Sourcing  Management 

1 1 :00  a.m. 

Panelists: 

Kris  Keener,  Assistant  Director,  GAO  Acquisition  and  Sourcing 
Management 

Travis  Masters,  Assistant  Director,  GAO  Acquisition  and  Sourcing 
Management 

Cheryl  Andrew,  Assistant  Director,  GAO  Acquisition  and  Sourcing 
Management 

Michael  Sullivan — is  the  Director,  Acquisition  and  Sourcing  Management,  U.S.  Government 
Accountability  Office.  This  group  has  responsibility  for  examining  the  effectiveness  of  the  DoD’s 
acquisition  and  procurement  practices  in  meeting  its  mission  performance  objectives  and 
requirements.  In  addition  to  directing  reviews  of  major  weapon  system  acquisitions  such  as  the  Joint 
Strike  Fighter,  F-22,  Global  Hawk,  and  various  other  major  weapon  acquisition  programs,  Sullivan 
has  developed  and  directs  a  body  of  work  examining  how  the  DoD  can  apply  best  practices  to  the 
nation’s  largest  and  most  technically  advanced  weapon  systems  acquisition  system.  This  work  has 
spanned  a  broad  range  of  issues  critical  to  the  successful  delivery  of  systems,  including  technology 
development,  product  development,  transition  to  production,  software  development,  program 
management,  requirement-setting,  cost  estimating,  and  strategic  portfolio  management.  The  findings 
and  recommendations  from  this  work  have  played  a  major  role  in  the  department’s  recent  acquisition 
policy  revisions.  Most  recently,  he  has  directed  the  GAO’s  annual  assessment  of  major  weapon 
systems  programs  for  Congress  and  the  GAO’s  work  with  Congress  in  establishing  acquisition  policy 
reforms.  His  team  also  provides  Congress  with  early  warning  on  technical  and  management 
challenges  facing  these  investments.  Sullivan  has  been  with  the  GAO  for  24  years.  He  received  a 
bachelor’s  degree  in  political  science  from  Indiana  University  and  a  master’s  degree  in  public 
administration  from  the  School  of  Public  and  Environmental  Affairs,  Indiana  University. 
[sullivanm@gao.gov] 

Kris  Keener — Assistant  Director,  GAO  Acquisition  and  Sourcing  Management 
Travis  Masters — Assistant  Director,  GAO  Acquisition  and  Sourcing  Management 
Cheryl  Andrew — Assistant  Director,  GAO  Acquisition  and  Sourcing  Management 
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Panel  2.  Applications  of  Real  Options  Analysis  in 
Defense  Acquisition 


Wednesday,  May  4,  2016 

11:15  a.m.  - 
12:45  p.m. 

Chair:  James  E.  Thomsen,  Former  Principal  Civilian  Deputy,  Assistant  Secretary 
of  the  Navy  for  Research,  Development,  &  Acquisition 

Acquiring  Technical  Data  With  Renewable  Real  Options 

Michael  McGrath,  ANSER 

Christopher  Prather,  Senior  Associate  Analyst,  ANSER 

Incorporation  of  Outcome-Based  Contract  Requirements  in  a  Real  Options 
Approach  for  Maintenance  Planning 

Xin  Lei,  Research  Assistant,  University  of  Maryland 

Navid  Goudarzi,  Postdoctoral  Researcher,  University  of  Maryland 

Amir  Reza  Kashani  Pour,  Research  Assistant,  University  of  Maryland 

Peter  Sandborn,  Professor,  University  of  Maryland 

Measuring  the  Return  on  Investment  and  Real  Option  Value  of  Weather 
Sensor  Bundles  for  Air  Force  Unmanned  Aerial  Vehicles 

Thomas  Housel,  Professor,  NPS 

Johnathan  Mun,  Research  Professor,  NPS 

David  Ford,  Research  Associate  Professor,  NPS 

Sandra  Horn,  Research  Associate,  NPS 

Dave  Harris,  NPS 

Matt  Cornachio,  NPS 

James  E.  Thomsen — former  principal  civilian  deputy,  assistant  secretary  of  the  navy  (ASN) 
research,  development  and  acquisition  (RDA).  As  the  principal  civilian  deputy  to  the  Honorable  Sean 
Stackley,  Thomsen’s  responsibilities  included  oversight  and  policy  for  Navy  and  Marine  Corps 
research,  development,  and  acquisition  programs  for  Shipbuilding,  Aviation,  Space,  and  Weapons 
systems.  In  his  capacity,  Thomsen  was  responsible  for  more  than  $100  billion  annually;  hundreds  of 
technical  development  and  procurement  programs  for  the  Department  of  the  Navy;  and  the 
department’s  Senior  Executive  Acquisition  Corps.  Thomsen  also  served  as  co-executive  director  for 
the  Under  Secretary’s  Better  Buying  Power  initiative  for  defense  acquisition  from  2010  to  2015. 

Thomsen  also  served  as  the  Program  Executive  Officer  (PEO)  for  Littoral  and  Mine  Warfare  (LMW), 
as  well  as  Executive  Director  of  the  PEO  (LMW).  As  the  PEO,  Thomsen  was  responsible  for  the 
execution  of  more  than  $3  billion  annually  on  technical  programs  that  included  Counter-IED  Electronic 
Warfare  Systems  in  response  to  Operation  Iraqi  Freedom  and  Operation  Enduring  Freedom;  ASW, 
SuW,  and  MIW;  Mission  Modules  for  the  Littoral  Combat  Ship;  Special  Warfare  Operations  Systems; 
Anti-Terrorism  Naval  Ship  Systems;  and  all  Naval  Undersea  Surveillance  Systems. 

Thomsen  has  held  several  technical  and  management  positions  within  the  Navy’s  Engineering 
Commands — Department  Head  of  Naval  Weapons  Systems  at  Dahlgren,  VA;  Department  Head  of 
Coastal  Warfare  Systems  at  Panama  City,  FL;  and  Program  Manager  of  Undersea  Weapons  at  the 
Naval  Sea  Systems  Command  in  Washington,  DC. 
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Thomsen  received  his  bachelor’s  degree  in  ocean  engineering  in  1981  from  Florida  Atlantic  University 
and  his  master’s  degree  from  Florida  State  University. 

Thomsen  has  received  numerous  career  awards  for  his  work  in  Defense  and  R&D  programs.  For  his 
career  achievements  in  the  Department’s  Senior  Executive  Corps,  Thomsen  was  awarded  the 
Presidential  Rank  Award  (Distinguished)  in  2010,  as  well  as  the  Secretary  of  Defense  Distinguished 
Performance  Medal  in  2012. 
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Acquiring  Technical  Data  With  Renewable  Real  Options 


Michael  McGrath — holds  a  BS  in  space  science  and  applied  physics  and  an  MS  in  aerospace 
engineering  from  Catholic  University,  and  a  doctorate  in  operations  research  from  George 
Washington  University.  As  a  former  Vice  President  at  ANSER,  he  led  business  operations  in  systems 
and  operations  analysis.  He  previously  served  as  Deputy  Assistant  Secretary  of  the  Navy  for 
Research,  Development,  Test,  and  Evaluation,  and  is  a  strong  proponent  for  improvements  in 
technology  transition,  modeling  and  simulation,  and  test  and  evaluation.  His  research  interests  are  in 
cybersecurity  for  manufacturing  and  procurement  and  use  of  digital  technical  data. 
[michael.mcgrath@anser.org] 

Chris  Prather — holds  a  BA  in  sociology  from  the  University  of  Washington  and  a  dual-title  MA  in 
sociology  and  demography  from  Pennsylvania  State  University.  He  is  currently  working  as  an  analyst 
in  the  Threat  &  Risk  Analysis  Division  at  the  Homeland  Security  Studies  &  Analysis  Institute,  a 
federally  funded  research  and  development  center  operated  by  ANSER  on  behalf  of  DHS.  At  ANSER, 
he  has  worked  on  a  variety  of  projects  involving  demographic  methodology  and  advanced  statistics, 
with  a  particular  focus  on  quantitative  methodology.  His  interests  include  policy  analysis,  methodology 
development,  and  forecasting,  [christopher.prather@hsi.dhs.gov] 

Abstract 

This  paper  investigates  the  use  of  real  options  as  a  strategy  to  hedge  risks  in  situations 
where  the  need  for  contract  deliverables  is  uncertain  over  a  long  life  cycle.  It  focuses  on  the 
case  of  contracting  for  technical  data  to  support  competitive  spares  procurement,  and  it 
proposes  a  data  maintenance  contract  with  renewable  options  to  deliver  technical  data  at  a 
pre-negotiated  price  at  the  time  of  need  and  the  required  level  of  data  rights.  A  business  case 
analysis  tool  is  developed  using  dynamic  programming  to  calculate  the  value  of  the  technical 
data  options  to  the  government.  This  tool  is  applied  in  an  example  using  available  cost  data 
to  support  a  series  of  annual  decisions  on  whether  to  continue  the  option,  and  to  determine 
the  optimal  timing  to  exercise  the  option  to  rent  or  buy  the  technical  data  based  on  the 
expected  cost  avoidance  to  the  government.  This  options-based  approach  helps  the 
government  avoid  the  costly  acquisition  of  technical  data  that  may  never  be  used  while 
ensuring  data  are  available  when  a  need  arises.  Industry  also  benefits  from  the  data 
maintenance  contract  as  a  business  opportunity  that  provides  more  accurate  data  for  system 
support  and  better  insight  into  government  uses  of  the  data. 

Introduction 

Acquisition  program  managers  are  expected  to  acquire  technical  data  needed  for  life 
cycle  sustainment  functions  such  as  maintenance  or  competitive  spare  parts  procurement, 
but  this  expectation  is  more  complicated  than  it  seems  (DoD,  2015).  The  needs  and  timing 
for  competitive  spare  parts  procurement  are  uncertain,  and  changes  in  system  configuration 
or  sustainment  strategy  can  alter  the  need  for  technical  data.  Additionally,  price  negotiation 
for  the  technical  data  package  (TDP)  often  occurs  in  a  sole  source  environment,  with 
conflicting  assertions  by  the  contractor  and  government  over  rights  in  data,  an  issue  that  is 
compounded  by  inadequate  business  case  evaluations  of  the  value  of  the  data  to  the 
government  (DoD,  1993).  In  some  instances,  prices  in  excess  of  $1  billion  have  been 
quoted  for  the  acquisition  of  TDPs  (GAO,  201 1 ).  Consequently,  TDPs  that  are  needed  are 
often  not  acquired,  TDPs  that  are  acquired  are  often  not  properly  priced,  and  TDPs  that  are 
delivered  may  never  be  used.  Program  managers  need  better  ways  to  hedge  uncertainty  in 
technical  data  needs  and  better  business  case  analysis  tools  for  the  procurement  of  TDPs. 

This  research  investigates  a  new  method  for  acquiring  technical  data  with  flexible 
options  to  be  exercised  at  the  time  of  need  during  the  product  life  cycle.  The  option  would 
allow  the  government  the  right,  but  not  the  obligation,  to  rent  or  purchase  the  technical  data 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-5- 


and  technical  data  deliverables  at  the  time  the  data  are  needed.  Purchasing  an  option 
preserves  the  opportunity  to  acquire  technical  data  deliverables  at  a  set  price  while  hedging 
the  risk  that  the  technical  data  ultimately  may  not  be  needed.  Because  the  data  are  not 
acquired  until  the  time  of  need,  this  helps  to  ensure  that  the  associated  data  rights  are 
acquired  at  the  appropriate  level  for  the  intended  technical  data  use.  This  allows  program 
managers  the  ability  to  continuously  reassess  needs  and  mitigate  changes  in  supply  chain, 
system  configuration,  or  sustainment  strategy. 

To  calculate  the  value  of  an  option  to  the  government  for  the  purchase  of  technical 
data  rights  and  deliverables,  we  use  real  options  theory,  which  accounts  for  the  costs  or 
savings  associated  with  various  alternative  outcomes.  Real  options  theory  originated  from 
the  valuation  of  options  in  the  financial  market.  Instead  of  valuing  the  option  to  purchase  a 
stock,  however,  real  options  theory  extends  this  valuation  to  the  purchase  of  “real”  things 
such  as  technical  data  packages,  which  we  explore  in  detail.  We  use  dynamic  programming 
to  value  the  real  option,  and  package  the  valuation  algorithm  in  a  user-friendly  Excel-based 
business  case  analysis  tool  that  is  freely  available.  We  present  a  proposed  business  model 
for  how  to  use  this  business  case  analysis  tool  in  a  real-world  scenario. 

Although  there  are  many  government  needs  for  technical  data  (engineering 
investigations,  depot  maintenance,  spares  procurement,  etc.)  we  limit  our  focus  in  this 
research  to  TDPs  and  associated  data  rights  used  in  competitive  procurement  of  spares  and 
repair  parts.  A  complete  TDP  will  cover  all  the  parts  in  a  system  or  subsystem.  Although 
spares  (repairable  items)  and  repair  parts  (consumable  items)  are  managed  differently  in  the 
DoD  supply  system,  there  is  no  difference  from  the  standpoint  of  TDP  data  deliverables 
needed  to  support  competitive  procurement.  So,  for  simplicity,  we  will  use  the  term  spare 
parts  to  include  both  categories.  To  illustrate  the  decision  support  tool  proposed  for  the  new 
acquisition  approach,  we  use  a  scenario  involving  the  data  deliverables  and  data  rights 
needed  for  competitive  procurement  of  a  single  part  numbered  item. 

Current  Acquisition  Policy  and  Practice 

DoD  acquisition  policy  requires  the  acquisition  program  manager  to  consider 
procuring  technical  data  and  associated  data  rights  during  acquisition.  Implicitly  underlying 
this  policy  is  an  expectation  that  the  acquisition  cost  of  technical  data  will  be  more  than 
offset  by  the  downstream  benefits  of  competition  and  other  benefits  of  DoD  use  of  the  data. 
DoD  Instruction  (DoDI)  5000.02  (DoD,  2015)  requires  that 

program  management  must  establish  and  maintain  an  IP  [Intellectual 
Property]  Strategy  to  identify  and  manage  the  full  spectrum  of  IP  and  related 
issues  (e.g.,  technical  data  and  computer  software  deliverables,  patented 
technologies,  and  appropriate  license  rights)  from  the  inception  of  a  program 
and  throughout  the  life  cycle. 

This  requirement  was  strongly  re-emphasized  in  the  DoD’s  Better  Buying  Power  2.0 
(BBP  2.0)  initiative  as  a  means  to  ensure  that  the  DoD  is  positioned  for  competitive  sourcing 
of  materials  needed  for  sustainment  and  upgrades  to  the  system  (OUSD[AT&L],  2012).  As  a 
result  of  BBP  2.0,  the  DoD  published  a  Data  Rights  brochure,  updated  DoDI  5010.12M  on 
Procedures  for  the  Acquisition  and  Management  of  Technical  Data,  and  developed  an 
Intellectual  Property  Strategy  Guidance  brochure  to  support  data  rights  planning.  Army, 
Navy,  and  Air  Force  documents  provide  further  guidance  on  the  acquisition  of  the  data 
deliverables  that  comprise  a  TDP.  Technical  data  is  a  significant  area  of  emphasis  in  DoD 
acquisition  policy;  Federal  Acquisition  Regulations  provide  standard  contract  requirements 
for  acquisition  of  technical  data  and  associated  IP  rights.  MIL  STD  31000a  prescribes  the 
content  of  TDPs  and  TDP  data  management  products,  and  the  DoD  acquisition  workforce  is 
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trained  in  assessing  technical  data  needs,  conducting  business  case  analyses  on  technical 
data  acquisition  strategies,  and  contracting  for  data  and  data  rights. 

In  practice,  however,  it  is  difficult  to  determine  life  cycle  data  needs,  evaluate  the 
business  case,  negotiate  and  contract  for  priced  data  rights  and  deliverables,  validate 
deliverables,  maintain  technical  data,  and  make  the  data  accessible  for  use  over  an 
extended  period.  Additionally,  industry  is  reluctant  to  release  technical  data  that  may  be 
used  by  potential  competitors.  There  may  also  be  circumstances,  such  as  contractor 
maintenance  of  the  system  under  a  Performance  Based  Logistics  (PBL)  arrangement, 
where  the  government  may  not  need  the  data  during  a  specified  period,  but  may  need  the 
data  later  to  maintain  a  competitive  market.  Given  the  uncertainty  of  needs  and  the  difficulty 
and  expense  of  procurement,  technical  data  are  often  deferred  or  put  in  a  contract  option 
that  is  never  exercised.  The  Government  Accountability  Office  (GAO)  has  published  several 
reports  critiquing  the  Department’s  handling  of  technical  data  acquisition.  In  2004,  the  GAO 
reported  that  “program  managers  often  opt  to  spend  limited  acquisition  dollars  on  increased 
weapon  system  capability  rather  than  on  acquiring  rights  to  the  technical  data — thus  limiting 
their  flexibility  to  perform  maintenance  work  in  house  or  to  support  alternative  source 
development  should  contractual  arrangements  fail.”  In  2010,  the  GAO  reported  “For  27  of 
the  47  noncompetitive  DoD  contracts  we  reviewed,  the  government  was  unable  to  complete 
requirements  due  to  a  lack  of  access  to  proprietary  technical  data.”  More  recently,  the  GAO 
(2011)  reported  that,  although  DoD  policies  have  been  updated  to  require  determination  of 
data  needs,  business  case  analysis  and  inclusion  of  technical  data  and  data  rights  in  the 
acquisition  strategy,  these  policies  are  sparsely  implemented  in  the  acquisition  programs 
they  reviewed.  The  disconnect  between  technical  data  acquisition  policy  and  practice  has 
been  a  longstanding  issue  in  the  DoD. 

In  the  section  titled  A  New  Acquisition  Strategy  for  Technical  Data,  we  propose  a 
new  acquisition  approach  designed  to  address  these  pragmatic  difficulties  by  creating  and 
preserving  competitively  priced  options  for  deferred  delivery  of,  or  access  to,  technical  data 
at  the  time  of  need  throughout  the  life  cycle.  This  approach  is  motivated  not  only  by  the 
need  to  reconcile  policy  and  practice,  but  also  by  the  opportunity  to  take  advantage  of 
technology  trends  affecting  technical  data. 

Technology  Trends  Affecting  Technical  Data 

Two  important  industry  trends  are  changing  DoD  practices  for  acquiring  and  using 
TDPs:  3-dimensional  (3D)  digital  product  models  and  product  life  cycle  management  (PLM) 
systems. 

3D  digital  product  models  have  revolutionized  industry  engineering  practices,  and 
are  now  affecting  DoD  practices.  When  DoD  policies  and  standards  for  TDPs  were  originally 
developed,  hard  copy  2D  engineering  drawings  produced  by  draftsmen  were  the  norm. 
These  drawings  required  interpretation  by  skilled  machinists  to  produce  a  part.  The  broad 
adoption  in  the  1980s  of  computer  aided  design  (CAD),  computer  aided  engineering  (CAE), 
and  computer  aided  manufacturing  (CAM)  systems  shifted  this  paradigm.  Today,  the 
aerospace  and  defense  industries  use  CAD/CAE  systems  to  generate  engineering  data  in 
digital  form,  often  called  the  “digital  thread”  or  “digital  tapestry”  that  drives  modeling, 
analysis,  and  automated  processes  throughout  the  manufacturing  enterprise  (Model  Based 
Enterprise,  2016).  The  DoD  is  gradually  equipping  itself  to  acquire  and  use  3D  digital  data  in 
engineering,  maintenance,  and  supply  applications.  The  advantages  of  a  3D  TDP  for  spares 
procurement  were  demonstrated  in  a  recent  Manufacturing  Technology  program  (U.S.  Army 
Armament  Research,  Development,  and  Engineering  Center,  2009).  Faced  with  diminishing 
sources  for  M2  .50  caliber  machine  gun  parts,  an  Army  engineering  center  entered  the  old 
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2D  drawings  into  a  CAD  system,  generated  a  3D  TDP,  and  prototyped  the  part  to  capture 
the  manufacturing  recipe.  When  the  validated  3D  TDP  and  manufacturing  process  data  files 
were  released,  bids  were  received  from  new  suppliers  who  said  they  would  not  have  bid 
without  the  digital  files.  The  parts  were  ultimately  delivered  with  a  70%  savings  in 
manufacturing  run  time  and  a  45%  savings  in  cost  compared  to  prior  procurements.  The 
conclusion  is  that  the  value  to  the  government  of  a  TDP  used  for  spares  procurement 
increases  when  the  TDP  is  available  in  a  3D  format.  Other  government  users  of  TDPs  in 
engineering  and  maintenance  organizations  have  similarly  concluded  that  3D  TDPs  add 
considerable  value.  Recognizing  the  value  of  3D  TDPs,  the  DoD  has  issued  a  new  standard 
practice  for  acquiring  either  2D  or  3D  TDPs  (DoD,  2013).  For  the  technical  data  acquisition 
approach  proposed  in  this  research,  we  assume  the  government  will  prefer  delivery  of  3D 
TDPs. 


PLM  systems  are  a  more  recent  development  in  industry,  but  have  grown  rapidly, 
reaching  $40  billion  in  global  sales  in  2014.  An  article  in  PR  Newswire  recognized  this  rapid 
growth,  noting  that 

aerospace  and  defense  was  the  largest  end-use  segment  of  the  PLM  market 
in  2014.  This  segment  has  a  significantly  long  product  development  cycle  and 
in  order  to  manage  this,  the  companies  in  this  sector  started  adopting  PLM 
solutions  in  wide  manner.  (Wood,  2015) 

The  primary  function  of  a  PLM  system  is  to  manage  product  information  of  all  types 
used  in  engineering,  manufacturing,  product  support,  and  business  processes  throughout 
the  life  cycle.  Product  information  starting  in  the  conceptual  phase  is  developed  in  a 
distributed  collaborative  environment,  linked,  configuration-managed,  and  made  accessible 
to  downstream  users  for  re-use  without  duplicating  the  data  or  allowing  it  to  get  out  of  synch. 
The  value  to  industry  stems  from  the  ability  of  PLM  systems  to  reduce  time  and  errors 
associated  with  locating  complex  data  sets  and  reconciling  version  control  issues. 
Government  organizations  see  potential  value  in  using  PLM  systems  to  archive  and  manage 
technical  data  delivered  to  the  government.  Naval  Air  Systems  Command,  for  example,  is 
reviewing  the  capabilities  of  systems  offered  by  major  PLM  vendors  with  a  view  toward 
procuring  such  a  system  (Owens  &  Gordon,  2014).  Some  PLM  systems  enable  trusted 
partners  to  share  access  to  a  PLM  database  and  associated  CAD  systems,  to  export  data 
sets  from  one  PLM  system  for  ingestion  into  another  PLM  system,  or  to  create  digital  files 
(e.g.,  a  3D  TDP)  for  transmission  to  users  who  have  no  PLM  access  (Doyle  &  Grossman, 
2014).  Such  systems  typically  include  strong  digital  rights  management  features  that  are 
suitable  for  protecting  intellectual  property  in  both  commercial  and  government  uses.  The 
technical  data  acquisition  approach  in  the  next  section  assumes  that  in  the  future,  contractor 
and  government  organizations  will  use  PLM  systems  to  manage  technical  data  for  speed 
and  accuracy  in  the  generation  of  a  bill  of  materials,  2D  drawings  and  3D  product  models, 
supporting  engineering  analysis  data,  manufacturing  process  and  tooling  data,  and 
numerous  other  types  of  data. 

A  New  Acquisition  Strategy  for  Technical  Data 

As  described  by  the  GAO  (201 1 ),  the  current  acquisition  approach  includes  the 
following  four  phases: 

1 .  Requirements,  strategies,  and  plans  phase — the  government  determines 
needs  for  technical  data  and  data  rights  and  includes  those  requirements  in 
the  acquisition  strategy  and  plan. 
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2.  Contracting  phase — the  government  specifies  data  requirements  in  the 
solicitation,  evaluates  competitive  contractor  proposals,  negotiates,  and 
awards  a  contract. 

3.  Contract  performance  and  delivery  phase — the  contractor  develops  and 
delivers  (or  provides  access  to)  technical  data  per  the  contract.  Delivery  may 
be  deferred  at  the  government’s  option  for  up  to  three  years  after  the  end  of 
the  contract  (DFARS  227.71).  Ultimately,  the  government  accepts  delivery  of 
the  data  into  a  government  storage  and  distribution  system. 

4.  Post-performance  and  sustainment  phase — the  government  uses  the 
technical  data  in  engineering,  maintenance,  supply  support,  and  other  life 
cycle  functions. 


The  proposed  new  method  uses  the  same  four  phases,  but  adds  flexibility  by  using  a 
subscription  to  the  contractor  PLM  system  for  online  access  and  options  for  deliverables  to 
hedge  risks  and  uncertainties  in  life  cycle  needs  for  technical  data.  Key  differences  include 
the  following: 


•  Needs  determination  is  essentially  the  same  in  Phase  1 ,  but  a  new  business 
case  analysis  tool  (described  in  the  following  section)  is  used  in  developing 
an  options-based  acquisition  strategy.  This  tool  considers  the  value  to  the 
government  of  having  the  option,  at  any  point  in  the  life  cycle,  to  access 
technical  data  maintained  by  the  contractor,  rent  TDPs  for  one-time  use,  or 
deliver  TDPs  to  a  government  system. 

•  In  Phase  2,  the  solicitation  requires  online  access  through  a  subscription  to 
the  contractor’s  PLM  system  (with  appropriate  data  rights)  during  the  contract 
period,  and  competitively  priced  options  for  delivery  or  one-time  use  (rental) 
of  TDPs  that  may  be  exercised  up  to  three  years  after  the  end  of  the  contract 
using  a  standard  DoD  contract  clause  (FAR  Parts  204,  212,  and  252). 

•  In  Phase  3,  the  government  has  the  option  to  accept  delivery  of  data,  but 
relies  primarily  on  access  to  the  contractor  PLM  system.  For  data 
deliverables  this  is  similar  to  the  deferred  ordering  clause  in  DoD  5010.12-M, 
which  “ensures  the  availability  of  the  raw  data  while  avoiding  the  cost  of 
buying  the  data,  if  the  need  never  arises.”  The  proposed  framework  differs  in 
that  all  data  deliverables  are  priced  during  Phases  2  and  3  at  the  appropriate 
level  of  rights.  In  contrast,  the  deferred  ordering  clause  pertains  only  to  items 
developed  at  government  expense,  in  which  case  “the  contractor  is 
compensated  only  for  the  cost  of  converting  the  technical  data  or  software 
into  the  required  format  and  for  reproduction  and  delivery.”  Also  in  Phase  3, 
the  government  plans  and  negotiates  a  sole  source  follow-on  data 
maintenance  contract  for  award  before  the  base  contract  data  options  expire. 
This  sole  source  negotiation  is  bounded  by  the  fact  that  the  government  can 
exercise  the  prior  competitively  priced  option  for  delivery  of  all  the  data  if  the 
proposed  price  of  follow-on  data  maintenance  is  too  high.  The  data 
maintenance  contract  also  includes  a  subscription  to  the  contractor  PLM 
system  that  may  be  renewed  as  needed  throughout  the  life  cycle. 

•  Finally,  in  Phase  4  the  government  meets  life  cycle  needs  either  by  using 
data  already  delivered  into  a  government  system  or  by  making  case  by  case 
decisions  at  the  time  of  need  on  whether  to  exercise  an  option  for  data 
delivery  or  one-time  use  (rental),  with  pricing  based  on  the  level  of  data  rights 
needed.  Figure  1  illustrates  the  data  flow  between  contractor  and  government 
systems  in  Phases  3  and  4. 
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Figure  1.  Modes  of  Data  Flow  Between  Contractor  and  Government  Systems 

The  major  effect  of  this  new  acquisition  approach  is  that  it  allows  the  government  to 
acquire  only  the  technical  data  needed,  with  the  data  rights  needed,  at  the  time  of  need 
rather  than  acquiring  all  the  data  during  acquisition  with  the  highest  level  of  data  rights.  In 
current  practice,  data  not  procured  during  acquisition  may  later  have  to  be  priced  and 
procured  in  a  sole  source  environment.  The  new  approach  preserves  option  prices  that  are 
competitively  priced  in  the  acquisition  phase. 

An  excellent  example  of  using  competition  for  leverage  on  pricing  is  the  Request  for 
Proposals  (RFP)  for  the  Joint  Logistics  Tactical  Vehicle  Family  of  Vehicles  (JLTV  FOV;  U.S. 
Army  Contracting  Command,  2014).  The  RFP  required  that  the  contractor  develop  and 
maintain  the  TDP  for  the  life  of  the  contract,  that  the  government  have  the  option  for 
purchase  and  delivery  of  the  TDP  at  a  firm  fixed  price,  and  that  the  contractor  warrant  the 
correctness  of  the  TDP.  The  government  required  that  the  3D  product  model  be  the  design 
master  record,  and  that  delivery  under  the  contract  option  use  a  government  PLM  system: 

The  Contractor  shall  perform  all  work  under  this  contract  using  the 
Government  Windchill  PDMLink,  beginning  with  the  date  the  Government 
exercises  the  TDP  Option  and  shall  provide  models  and  CAD  files  which 
successfully  pass  the  quality  checks  and  Windchill  PDMLink  release  process 
defined  in  these  modeling  standards. 

To  incentivize  delivery  of  a  complete  TDP,  the  RFP  included  a  novel  provision  that 
gave  the  offerors  credit  for  a  TDP  Adjustment  in  the  Total  Evaluated  Cost/Price  factor  in 
source  selection  (U.S.  Army  Contracting  Command,  2014).  The  TDP  Adjustment  was  based 
on  a  government  estimate  of  $51 1  million  in  expected  life  cycle  savings  if  the  TDP 
supported  future  full  and  open  competitive  acquisitions.  Credit  was  given  for  the  difference 
between  the  offeror’s  TDP  price  and  government  savings  estimate,  adjusted  by 
completeness  of  the  offered  TDP  and  data  rights.  The  three  offerors  responding  to  the  RFP 
were  incentivized  to  get  maximum  credit  by  offering  TDPs  with  no  restrictions  on  use  for 
competitive  re-procurement,  thereby  allowing  the  government  to  avoid  the  cost  of  reverse 
engineering  and  qualification  testing  for  secondary  sources. 

According  to  current  government  users  of  technical  data,  past  practice  in  exercising 
TDP  purchase  options  has  often  encountered  problems  in  the  timeliness  of  delivery, 
completeness,  and  accuracy  of  technical  data  deliverables.  Contractor  and  government 
PLM  systems  will  be  helpful  in  avoiding  past  problems  in  delivery  times,  in  review  of  data 
rights  markings,  and  in  configuration  accuracy  and  completeness  of  TDPs.  A  continued 
contractual  relationship  during  the  sustainment  phase  would  allow  the  government  to 
enforce  contract  requirements  more  easily.  In  the  new  acquisition  method,  provisions  of  the 
data  maintenance  contract  could  include  requirements  for  timeliness  of  delivery,  accuracy, 
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and  completeness  of  the  TDP  for  use  in  competitive  re-procurements,  and  specified  formats 
for  deliverables  suitable  for  government  repositories  or  PLM  systems. 

From  the  contractor  point  of  view,  the  subscription  to  the  contractor  PLM  system 
presents  a  new  business  opportunity  over  the  life  cycle.  Making  accurate,  up-to-date  data 
available  for  government  purposes  can  avoid  problems  for  the  contractor  as  well  as  the 
government.  Perhaps  the  greatest  benefit,  however,  is  the  ability  to  avoid  potential  delays  in 
production  decisions  by  agreeing  on  options  rather  than  relying  on  the  government  to  find 
full  funding  for  technical  data  acquisition  to  meet  acquisition  milestone  decision 
requirements. 

A  Decision  Framework  Based  on  Real  Options  Theory 

The  options-based  method  for  acquiring  technical  data  requires  government 
decisions  on  whether  to  contract  for  options,  whether  to  extend  options  by  renewing  the  data 
maintenance  contract,  and  whether/when  to  exercise  options  to  rent  or  buy  the  data  at  the 
appropriate  level  of  data  rights.  The  nature  and  timing  of  a  government  need  for  technical 
data  is  uncertain.  Therefore,  we  use  a  real  options  theory  approach  to  calculate  the 
expected  value  of  the  option  to  acquire  the  TDP  and  determine  the  optimal  time  to  exercise 
this  option. 

Real  Options  Theory 

Real  options  theory  grows  out  of  the  valuation  of  options  in  the  financial  market. 
There,  the  purchase  of  an  option  allows  the  purchaser  the  right,  but  not  the  obligation,  to 
buy  or  sell  a  stock  at  a  fixed  price.  The  decision  of  whether  to  purchase  the  option  is  based 
on  the  calculation  of  the  option’s  value  relative  to  the  cost  of  the  option  (Goudarzi  & 
Sandborn,  2015).  As  an  example,  imagine  a  stock  that  is  currently  trading  at  $80,  where  the 
cost  of  an  option  is  $15  for  a  one-year  option  to  purchase  the  stock  at  the  exercise  price  of 
$70.  If  you  purchase  the  option,  and  exercise  it  on  the  same  day,  the  payoff  would  be  $10 
for  the  stock,  but  the  cost  of  the  option  is  higher  than  this  payoff,  meaning  you  would  end  up 
losing  $5.  If  you  waited,  however,  and  the  value  of  the  stock  increased  to  $100,  you  could 
then  exercise  the  option  at  the  $70  exercise  price,  and  will  make  $15  ($100  current  trading 
value  -  $70  exercise  price  -  $15  option;  Leslie  &  Michaels,  1997). 

Real  options  theory  extends  this  logic  to  real  assets,  such  as  factories,  real  estate, 
mines,  and  intellectual  property  (Sick  &  Gamba,  2005).  In  real  options  terms  for  technical 
data,  the  purchase  of  an  option  allows  the  purchaser  the  right,  but  not  the  obligation,  to 
acquire  the  TDP  and  deliverables  at  a  fixed  price.  In  addition  to  addressing  the  question  of 
“What  should  I  pay  to  buy  the  option?”  real  options  theory  also  assists  in  determining  when 
the  option  should  be  exercised  (Goudarzi  &  Sandborn,  2015).  For  the  case  of  technical  data, 
we  use  real  options  theory  to  account  for  the  uncertainty  in  need  associated  with  spare  parts 
as  well  as  the  variability  in  costs  of  acquiring  the  parts.  Calculating  the  value  of  the  option  at 
various  stages  in  the  program  life  cycle  provides  the  program  manager  the  information 
necessary  to  purchase  only  the  technical  data  that  is  needed  at  the  time  that  it  is  needed, 
and  at  the  appropriate  level  of  rights,  avoiding  the  costly  acquisition  of  technical  data  that 
may  never  be  used,  or  the  acquisition  of  technical  data  at  a  level  of  rights  that  is  not 
necessary. 

The  traditional  method  to  value  stock  options  is  the  Black-Scholes  model,  proposed 
by  Black  &  Scholes  in  1973.  Variations  of  the  Black-Scholes  model  are  still  widely  used,  but 
the  basic  assumptions  of  the  model  generally  do  not  hold  for  the  valuation  of  real  options. 
The  Black-Scholes  model  makes  assumptions  about  constant  volatility  in  price,  normal 
distribution  of  returns  and  lognormal  distribution  of  underlying  asset  value — assumptions 
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that  do  not  fit  many  real  option  scenarios.  More  importantly,  the  Black-Scholes  model  was 
developed  to  value  a  European-style  option,  which  is  an  option  that  must  be  exercised  at  a 
fixed  point  in  time.  Real  options,  on  the  other  hand,  are  usually  better  conceptualized  as 
American-style  options,  which  can  be  exercised  at  any  point  in  time  over  the  life  of  the 
option  (Gilbert,  2004). 

To  calculate  the  value  of  our  real  options  for  the  purchase  of  technical  data  and  data 
rights,  we  structure  the  problem  as  an  American-style  option  that  can  be  exercised  at  the 
time  of  need,  but  must  be  renewed  on  a  scheduled  basis.  We  calculate  the  value  of  the 
option  to  the  government  based  on  the  year  by  year  probability  of  need  (Bayesian  prior 
probability)  and  an  evaluation  of  expected  cost  avoidance.  Essentially,  we  are  valuing  the 
benefit  of  avoiding  the  expenses  of  working  around  the  lack  of  technical  data  that  would  be 
necessary  had  the  TDP  not  been  available.  For  example,  lack  of  technical  data  might 
necessitate  sole  source  procurement  of  a  spare  part  from  the  original  supplier.  If  there  is  a 
25%  savings  associated  with  competitive  procurement  of  the  part,  this  savings  would  be  a 
source  of  cost  avoidance  to  the  government. 

Decision  Tree  for  Technical  Data  Options 

In  Phases  3  and  4  of  our  technical  data  acquisition  method,  there  are  two  recurring 
decisions  to  be  evaluated.  The  first  considers  whether  to  pay  to  keep  the  option  open  or 
allow  it  to  lapse.  The  second  considers  whether  to  exercise  the  option  (buy  or  rent  the 
technical  data)  at  the  time  a  need  occurs.  Both  decisions  are  based  on  the  expected  net 
cost  avoidance  associated  with  various  government  uses,  summed  across  the  remaining 
years  of  the  life  cycle.  We  can  represent  this  as  a  decision  tree,  as  shown  in  Figure  2,  that 
decides  each  year  (labeled  stage  s)  whether  to  renew  the  data  maintenance  contract  and 
data  delivery  options,  and  then  decides  during  the  year  whether  to  exercise  an  option  based 
on  operational  needs. 
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Data 

Maintenance 
Contract 
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Cancel 
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Expected  Net 
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in  Stage  s 
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Out-year 
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SO 


□  Decision  Node  Q  Outcome  Node 


Figure  2.  Iterative  Decision  Tree 

Decision  trees  are  evaluated  by  working  backward,  from  right  to  left.  For  simplicity, 
assume  that  this  subscription  only  contains  one  technical  data  deliverable,  and  consider  just 
decisions  that  occur  during  one  year  (stage  s).  For  the  buy  option  (top  branch  of  the 
decision  tree),  if  the  government  buys  the  technical  data,  there  is  a  cost  avoidance  in  the 
current  stage  (expected  net  cost  avoidance  in  stage  s),  and  in  all  subsequent  stages  in  the 
future  (expected  out-year  cost  avoidance),  since  the  technical  data  are  now  available  in  a 
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government  system  for  future  use.  For  the  rent  option,  because  the  technical  data  are 
rented  for  a  limited  time,  the  cost  avoidance  accrues  to  the  government  only  during  the 
rental  period  (expected  net  cost  avoidance  in  stage  s).  If  the  technical  data  are  neither 
bought  nor  rented  during  stage  s,  there  is  no  cost  avoidance.  The  value  of  renewing  the 
subscription,  then,  depends  on  which  decision  is  chosen  (buy,  rent,  neither)  and  on  the 
inherent  value  of  online  access  to  the  contractor  system  for  data  that  has  not  yet  been 
delivered. 

Since  we  are  working  backward,  and  this  is  a  multi-stage  (i.e.,  multi-year)  problem, 
we  set  stage  s  to  be  s  =  N-1  where  N  is  the  last  year  of  the  life  cycle,  and  work  backward 
from  there.  If  a  need  occurs  with  only  one  year  of  life  remaining,  only  one  year  of  cost 
avoidance  is  possible.  Assuming  it  is  less  expensive  to  rent  the  data  than  to  buy  it,  the 
decision  would  be  to  rent  the  data  or  to  do  without,  whichever  generates  the  larger  expected 
net  cost  avoidance.  If  we  know  the  probability  of  need  for  spare  parts  in  the  last  year  of  the 
life  cycle,  the  difference  in  cost  between  meeting  that  need  with  and  without  delivered 
technical  data,  and  the  cost  of  renting  the  technical  data,  we  can  compute  the  expected  net 
cost  avoidance  and  choose  the  optimal  path  for  that  year. 

In  similar  fashion  we  can  back  up  another  year  (s  =  N-2)  and  evaluate  expected  net 
cost  avoidance  for  each  branch  in  the  decision  tree.  We  compute  the  current  year  expected 
cost  avoidance  for  the  buy  and  rent  options,  and  for  the  buy  option  also  add  in  the  out-year 
cost  avoidance  calculated  in  the  previous  step.  We  continue  to  work  backward  to  the  current 
year,  always  choosing  the  decision  for  each  year  that  maximizes  cost  avoidance,  and 
recognizing  that  once  the  “buy”  decision  is  chosen,  all  remaining  out-years  benefit  from  the 
availability  of  technical  data.  This  results  in  an  optimal  path  through  the  many  branches  of 
the  multi-stage  decision  tree  shown  in  Figure  3.  The  example  scenario  discussed  below  will 
illustrate  one  such  optimal  path. 
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Figure  3.  Multi-Stage  Decision  Tree 

Formulation  as  a  Dynamic  Programming  Problem 

We  recognize  this  multi-stage  decision  problem  as  belonging  to  the  class  of  dynamic 
programming  problems  first  addressed  in  the  1950s  (Bellman,  1954).  To  find  the  series  of 
decisions  that  will  maximize  cost  avoidance,  we  define  the  following  variables: 

stage  s  =  [0,1,2  ...  N],  where  N  is  the  last  year  of  the  life  cycle 

decision  variable  Xj  =  [Buy,  Rent,  Neither]  for  j  =  [1,2,3] 


CSXj  =  Expected  net  cost  avoidance  in  stage  s  if  Xj  is  chosen  ( net  of  option  cost ) 
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C*s  =  max[/]  csx. 

CGSXj  —  Expected  gross  cost  avoidance  in  stage  s  if  Xj  is  chosen 

/(s,  *y)  =  total  cost  avoidance  of  best  policy  C sequence  of  choices ) 

for  the  remaining  stages  given  we  are  in  stage  s  and  choose  Xj 

We  maximize  total  cost  avoidance  by  starting  at  stage  N-1  and  working  backward, 
choosing  xyin  stage  s  that  maximizes: 


N  N 

f{s,Xj)  =  Cs  +  (1  -  Ss )  ^  Cl  +  Ss  ^  CGiXl 

i=s+ 1  i=s+ 1 


where  8S  =  1  if  C*s  =  Csx±  and  8S  =  0  otherwise 
since  "buy"  avoids  subsequent  gross  costs 

This  can  be  visualized  as  the  decision  tree  shown  in  Figure  4. 
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Figure  4.  Decision  Tree  Showing  Dynamic  Programming  Equation  to  Calculate 

Value  of  TDP  Option 

For  ease  of  computation,  we  developed  a  recursive  algorithm  to  evaluate  f(s,Xj)  for 
each  year  of  the  life  cycle,  starting  from  the  final  year: 

Let  OYSXl  =  outyear  gross  savings  if  is  chosen  in  period  s 

N 

OYSXl  =  ^  CGiXl 

i=s+ 1 


N 

Thenf(s,Xj)=  Cs*  +  (1  -  8S)  ^  C*  +  8s0YSXi 

i=s+ 1 
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Starting  at  s=N-1  and  working  backward, 

f(N  -  1  ,Xj)  =  Q_i  +  (1  -  SN_1)C^I  +  Sn_1OY(n_1)Xi 

f(N  -  2 ,Xj)  =  C*N_2  +  f(N  -  1  ,Xj) 

f(s,  Xj)  =  Cg  +  f(N  —  s  +  1,  Xj)  (This  is  a  recursive  algorithm) 

This  algorithm  has  been  implemented  in  an  Excel  spreadsheet  model  that  is  freely 
available.1  The  required  inputs  for  this  model  are:  the  year-by-year  probability  of  being  in  a 
buy  position  for  spare  parts,  the  forecasted  buy  quantity  of  parts  to  be  procured,  projected 
cost  resolution  data  if  TDP  is  not  available,  the  purchase  price  of  the  TDP,  the  rental  price  of 
the  TDP,  the  sole  source  price  of  a  single  spare  part  unit,  the  life  cycle  duration  for  which 
spare  parts  are  required,  and  the  discount  rate,  if  any,  to  be  applied  for  net  present  value 
calculations. 

With  this  business  case  analysis  tool,  for  any  given  spare  parts  acquisition  scenario, 
the  total  cost  avoidance  can  be  calculated  to  determine  the  initial  benefit  and  support  the 
decision  to  include  the  data  maintenance  and  data  delivery  option  line  items  during  initial 
acquisition.  The  decision  of  whether  to  continue  the  data  maintenance  and  delivery  options 
in  follow-on  contracts  can  be  evaluated  with  the  same  tool.  Finally,  the  tool  can  be  used  as 
needs  arise  during  the  life  cycle  to  decide  whether  to  buy  or  rent  the  technical  data  or  to 
meet  the  need  without  delivery  of  technical  data. 

Example  Scenario 

This  example  shows  how  the  calculations  might  apply  to  decisions  on  a  TDP  to 
support  spare  parts  procurement.  The  scenario  assumes  the  following: 

•  The  probability  of  being  in  a  spare  parts  buy  position  (p(spares))  in  any  given 
year  is  as  shown  in  Table  1 .2  When  spare  parts  are  procured,  the  buy 
quantity  is  always  a  lot  of  100. 

•  The  system  life  cycle  is  20  stages,  or  years.  A  contractor  PBL  program  is  in 
force  for  the  first  three  years  of  operation  (to  illustrate  how  options-based 
acquisition  of  data  could  complement  other  acquisition  practices). 

•  The  cost  of  the  subscription  is  zero.  In  practice,  the  cost  of  the  subscription 
would  be  significant  and  would  be  amortized  across  multiple  data 


1  Full  text  available  at 

http://anser.orq/docs/reports/Acquirinq  Technical  Data  with  Renewable  Real  Options.pdf; 

spreadsheet  model  available  at 

http://anser.org/docs/reports/Tech  Data  with  Real  Options  Spreadsheet  Model.xlsx 

2  Note  that  in  practice,  when  the  need  arises  for  spare  parts  procurement,  the  probability  of  being  in  a 
spare  parts  buy  position  becomes  1 .  The  probability  of  need  for  each  year  should  be  regularly  re¬ 
evaluated  based  on  changes  in  the  projected  forecast  for  spare  parts  procurement.  For  example, 
being  in  a  buy  position  in  one  year  might  increase  the  probability  of  being  in  a  buy  position  for  spare 
parts  in  subsequent  years.  The  probabilities  are  not  intended  to  remain  static  over  the  entire  life  cycle. 
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deliverables.  Since  this  scenario  looks  at  a  single  data  deliverable  and 
focuses  on  cost  avoidance  calculations,  we  omit  the  cost. 

•  The  TDP  data  deliverables  and  associated  rights  can  be  purchased  for 
$50,0003  or  rented  for  one  year  for  $5,000. 

•  Two  courses  of  action  are  available  when  the  TDP  is  not  delivered  to  the 
government:  sole  source  procurement  from  the  original  supplier,  or 
workarounds  to  enable  procurement  from  other  sources  without  a  TDP. 

o  If  spare  parts  are  purchased  in  a  sole-source  environment,  the  unit 
cost  is  $1,000.  If  they  are  sourced  competitively,  there  is  a  cost 
savings  of  25%,  for  a  unit  cost  of  $750  (Office  of  the  Inspector 
General,  1995). 

o  Workarounds  include  procuring  approved  substitutes,  qualifying  a  new 
substitute,  repair/refurbishment/reclamation,  reverse  engineering,  and 
redesign.  The  average  cost  of  these  workarounds  is  $159,179  for  our 
scenario.4  This  estimate  is  based  on  surveyed  cost  metric  data  from 
the  resolution  of  parts  obsolescence  problems  (Defense 
Standardization  Program  Office,  2015).  These  costs  can  be  avoided  if 
the  TDP  is  available  for  spares  procurement.  If  a  work-around  is 
implemented  for  a  particular  application,  the  out-year  costs  for  that 
application  become  zero. 


Table  1.  Probability  of  Being  in  a  Buy  Position  for  Spare  Parts  Procurement 


Stage 

1 

2 

3  4  5  6 

7  8  9  10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

p(spares) 

0 

0  .5  .5  .5 

5  5  4  4 

z 

a 

.3 

a 

.3 

.3 

a 

.2 

.1 

.01 

Using  these  assumptions,  the  recursive  algorithm  calculates  the  expected  cost 
avoidance  for  each  decision  at  each  time  point,  starting  at  year  20,  and  working  backwards 
to  year  1 .  At  each  time  point,  the  algorithm  selects  the  optimal  decision  (buy,  rent  or 
neither).  This  results  in  the  optimal  decision  path  shown  in  Table  2. 


Table  2.  Expected  Cost  Avoidance  for  Example  Scenario 


Expected  Cost  Avoidance  (thousands-  of  doDars) 

1 

2 

3 

4 

5 

6 

7 

8 
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10 

11 

12 

13  1  14  15  IS  1  17  1  13  1  19 
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Buy 

0 

0 

0 
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88 

72 

55  41  27  12  5  -IS  -34 

-45 

Rent 

0 

0 

0 

174 

162 

149 

136 

123 

110 

99 

8sl 

77| 

65|  57 

0  1 

Neither 

0 

0 

0 

-259 

-242 

-224 

-206 

-188 

-170 

-154 

-138 

-122 

-105  1-92  -77  |  -62  -45  -32  |  -16  | 

-5 

3  In  order  to  present  results  that  are  intuitively  clear,  we  set  the  discount  rate  to  zero  for  net  present 
value  calculations. 

4  The  $159,179  value  is  a  weighted  average  based  on  the  average  cost  of  each  workaround, 
weighted  by  the  probability  of  each  workaround  being  selected.  The  average  costs  for  each 
workaround  were  calculated  by  the  Defense  Standardization  Program  Office  based  on  responses 
collected  from  the  2014  Defense  Industrial  Base  Assessment:  Diminishing  Manufacturing  Sources 
and  Material  Shortages  Cost  Resolution  Values  Survey  conducted  by  the  Department  of  Commerce’s 
Bureau  of  Industry  and  Security. 
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In  the  first  three  years,  since  contractor  PBL  is  used,  the  probability  of  being  in  a  buy 
position  for  spare  parts  is  zero.  As  a  result,  in  these  years,  there  is  no  benefit  to  purchasing 
or  renting  a  TDP  for  spares  procurement. 

In  year  4,  if  the  government  were  to  exercise  the  option  to  buy  the  technical  data,  it 
would  accrue  $209,000  in  cost  savings  over  the  rest  of  the  life  cycle,  including  the  benefits 
in  the  current  year  and  all  expected  benefits  in  the  out-years.  Buying  the  TDP  continues  to 
be  the  optimal  decision  in  years  5  through  10.  In  year  1 1 ,  however,  the  expected  cost 
avoidance  for  buying  or  renting  the  technical  data  is  equal.  At  this  point,  the  combination  of 
low  probability  of  need  and  limited  remaining  years  of  benefit  make  it  equally  attractive  to 
meet  a  need,  if  one  occurs,  by  either  buying  or  renting  the  TDP.  In  year  12  and  beyond, 
renting  the  technical  data  becomes  the  optimal  decision. 

Monte  Carlo  Simulation 

The  business  case  analysis  tool  uses  expected  values  as  the  basis  for  decisions.  In 
order  to  evaluate  the  sensitivity  of  decisions  to  the  probability  of  being  in  a  buy  position  and 
variability  in  the  cost  metrics,  we  performed  two  separate  Monte  Carlo  simulations.  The  first 
used  a  uniform  distribution  to  vary  the  probability  of  being  in  a  spares  buy  position  between 
plus  or  minus  .05  of  the  values  reported  in  Table  1 .  The  results  of  1 ,000  runs  are  presented 
in  Figure  5. 


Stage  (years) 

Figure  5.  Monte  Carlo  Results  for  Varying  Probability  of  Buy  Position  for  Spares 

Figure  5  shows  the  expected  cost  avoidance  associated  with  buying  the  technical 
data  at  each  stage,  represented  by  the  green  lines,  and  renting  the  technical  data, 
represented  by  the  blue  lines.  The  solid  lines  show  the  mean  expected  cost  avoidance  at 
each  stage.  The  dashed  lines  of  each  color  above  their  mean  represent  the  expected  value 
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if  the  probability  of  being  in  a  buy  position  were  up  to  one  standard  deviation  higher  than  the 
mean,  and  dashed  lines  of  each  color  below  their  mean  represent  the  expected  value  if  the 
probability  of  being  in  a  buy  position  were  up  to  one  standard  deviation  lower  than  the  mean. 
The  resultant  bands  show  that  at  the  beginning  of  the  life  cycle  and  at  the  end  of  the  life 
cycle,  as  the  expected  cost  avoidance  values  diverge,  the  decision  to  rent  or  buy  is  less 
sensitive  to  variation  in  the  probability  of  being  in  a  buy  position.  Near  the  middle  of  the  life 
cycle,  however,  as  the  expected  cost  avoidance  values  for  buying  and  renting  the  TDP 
converge,  the  decision  to  rent  or  buy  is  more  sensitive  to  variation  in  the  probability  of  being 
in  a  buy  position.  This  shows  that  as  the  expected  value  for  renting  or  buying  the  TDP 
becomes  more  equal,  it  is  especially  important  to  have  an  accurate  assessment  of  the 
probability  of  being  in  a  buy  position. 

The  second  Monte  Carlo  simulation  allowed  the  resolution  cost  metrics  to  randomly 
vary  around  the  mean  according  to  a  normal  distribution  bounded  by  the  95%  confidence 
interval  reported  in  the  Diminishing  Manufacturing  and  Material  Shortages  report  (Defense 
Standardization  Program  Office,  2015).  Similar  results  to  Figure  5  were  obtained.  As  the 
expected  cost  avoidance  for  buying  versus  renting  the  technical  data  converges  in  the 
middle  years  of  the  life  cycle,  the  decision  is  more  sensitive  to  the  variation  in  the  resolution 
cost  metrics.  These  two  Monte  Carlo  simulations  show  that  in  the  middle  of  the  life  cycle, 
accurate  data  on  the  probability  of  being  in  a  buy  position  for  spares  and  cost  metrics  are 
essential  in  order  to  reduce  the  variation  in  the  estimates  and  make  a  more  accurate 
decision  to  buy  or  rent  the  TDP.  In  the  beginning  and  end  stages  of  the  life  cycle,  an 
accurate  decision  can  be  made  even  with  a  higher  variance  in  both  the  probability  of  being 
in  a  buy  position  and  cost  metrics  for  various  resolution  alternatives. 

Conclusion 

We  have  proposed  a  new  method  of  contracting  for  technical  data  using  options,  and 
a  business  case  analysis  tool  for  decision  support  in  acquisition  and  sustainment  phases.  In 
addition,  we  have  identified  the  contracting  issues  to  be  addressed  in  both  acquisition  and 
sustainment  phases,  the  opportunities  to  take  advantage  of  technology  trends  in  industry, 
and  the  potential  for  cost  avoidance  in  situations  where  government  needs  are  uncertain. 
Finally,  building  upon  the  basic  underlying  decision  tree  that  is  present  in  most  real  options 
settings,  we  have  developed  and  demonstrated  a  business  case  analysis  tool  using  a 
dynamic  programming  solution  algorithm.  The  business  case  analysis  tool  fits  cases  where 
the  timing  of  need  is  uncertain,  thereby  avoiding  the  restrictive  assumptions  of  the  traditional 
Black-Scholes  model.  The  Monte  Carlo  analysis  available  in  the  tool  can  be  used  to  test 
sensitivity  to  assumptions  and  interactions  among  variables. 

While  the  new  acquisition  method  is  applicable  for  technical  data  and  data  rights 
acquisition  to  meet  the  full  range  of  government  needs  for  technical  data,  we  have  illustrated 
its  application  in  only  a  single  scenario — TDPs  for  competitive  procurement  of  spares  and 
repair  parts.  Further  research  could  extend  the  business  case  analysis  to  other  government 
application  areas,  such  as  engineering  analysis,  weapon  system  upgrades,  and  depot 
maintenance.  The  underlying  decision  support  process  would  be  the  same  for  other 
application  areas,  but  the  probability  of  need  and  cost  avoidance  data  sources  would  differ. 

Our  research  was  limited  by  the  lack  of  publically  available  data.  Discussions  with 
DoD  practitioners  during  the  course  of  the  research  indicated  that  the  year-by-year 
probability  of  need  could  be  estimated  through  a  combination  of  reliability  data,  parts  usage 
data,  and  expert  opinion.  Cost  data  associated  with  courses  of  action  with  and  without 
availability  of  technical  data  are  also  available  within  the  DoD,  as  reflected  in  the  JLTV 
example  cited  where  a  government  estimate  of  $51 1  million  was  given  for  expected  life 
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cycle  savings  if  the  TDP  supported  future  full  and  open  competitive  acquisitions.  We  were 
told  by  both  government  and  industry  representatives  that  the  proposed  acquisition  method 
has  real  potential  for  use  and  may  merit  demonstration  in  a  pilot  program. 

Ideally,  a  pilot  program  would  have  an  established  baseline  for  comparison  of  the 
new  method  to  prior  methods,  and  it  would  be  executed  on  a  time  scale  of  tens  of  months 
rather  than  tens  of  years.  A  weapon  system  upgrade  program  might  be  suitable  as  a 
candidate  pilot  in  follow-on  research.  Key  elements  to  be  developed  or  investigated  in  such 
a  pilot  program  might  include  the  following: 

•  Solicitation  and  contract  language  to  incentivize  competitive  pricing  of 
technical  data  options 

•  Identification  of  data  sources  for  the  business  case  analysis  in  application 
areas  of  interest 

•  Provisions  for  government  online  access  to  contractor  PLM  systems,  and  for 
maintaining  and  synchronizing  technical  data  held  in  separate  government 
and  industry  systems 

•  Documentation  of  costs  and  savings  compared  to  prior  costs  for  data 
deliverables  and  data  rights  on  the  system  being  upgraded 

•  Evolution  of  the  business  case  analysis  tool,  its  connection  to  data  sources 
and  its  user  interface 

Finally,  we  note  that  real  options  are  widely  used  as  a  hedging  strategy  in  the 
investment  sector,  but  are  rarely  used  in  government  procurements  at  federal,  state,  or  local 
levels.  The  methods  and  models  developed  in  this  NPS-sponsored  research  are  now  freely 
available 

(http://anser.orq/docs/reports/Tech  Data  with  Real  Options  Spreadsheet  Model.xlsx)  and 

applicable  to  other  procurement  settings  where  the  public  has  a  long-term  interest  in 
sustainment  of  a  capability  and  a  need  to  mitigate  the  cost  and  risk  of  being  dependent  on  a 
sole  source  for  the  life  of  the  system. 
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Abstract 

Performance-based  logistics  (PBL)  is  growing  in  popularity  for  both  governmental  and  non¬ 
governmental  acquisitions  of  critical  systems.  These  contracts  allow  the  customer  to  buy  the 
performance  of  the  system  rather  than  purchase  the  system  and/or  to  buy  the  availability  of 
the  system  rather  than  pay  for  maintenance.  Outcome-based  contracts,  which  include  PBL, 
are  highly  quantified  “satisfaction  guaranteed”  contracts  where  “satisfaction”  is  defined  by  the 
outcomes  received  from  the  system  (i.e.,  the  specified  performance  level  or  availability). 

Maintenance  planning  seeks  to  predict  and  optimize  when  maintenance  for  a  system  is 
performed.  Condition-monitoring  technologies  such  as  Condition-Based  Maintenance  (CBM) 
and  Prognostics  and  Health  Management  (PHM)  provide  Remaining  Useful  Life  (RUL) 
estimates  that  can  be  used  to  plan  maintenance.  The  challenge  is  how  to  use  the  predicted 
RULs  (with  their  associated  uncertainties)  and  the  performance  requirements  imposed  by  the 
outcome-based  contracts  to  optimally  plan  future  maintenance. 

This  research  addresses  the  incorporation  of  outcome-based  contract  requirements  within  a 
real  options  approach  used  to  optimize  maintenance  planning.  A  simulation-based  real 
options  analysis  (ROA)  approach  is  used  to  determine  the  optimum  predictive  maintenance 
opportunity  for  a  system  managed  via  an  outcome-based  contract. 

Introduction 

Background  and  Motivation 

While  researchers  have  studied  planning  and  decision  making  for  outcome-based 
contracting  in  different  areas  (e.g.,  supply  chain,  logistics,  and  inventory  management)  and 
for  different  applications  (e.g.,  defense,  avionics,  railroads,  infrastructure,  and  energy),  there 
is  little  formal  work  dedicated  to  contractual  design  and  requirements  optimization  (Kashani- 
Pour  &  Sandborn,  2016). 

The  impact  of  contract  oriented  design  processes  on  original  equipment 
manufacturer  (OEM)  decision  making  for  optimizing  reliability  in  the  post-production 
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purchase  period  led  to  the  development  of  integrated  schemes  with  dynamic 
interdependencies  of  product  and  service,  called  product-service  systems  (PSSs;  Meier, 

Roy,  &  Seliger,  2010).  Procurement  and  system  acquisition  process  efficiency  and  success 
across  a  system’s  life  cycle  requires  the  development  and  implementation  of  best-value, 
long-term,  performance-based  product  support  strategies  that  leverage  performance-based 
agreements  with  both  industry  and  government  product  support  providers  (Datta  &  Roy, 
2010).  Hence,  an  effective  combination  of  technical  and  monetary  approaches  that  includes 
the  inventory,  maintenance,  and  operational  decisions  together  to  form  a  unified  model  that 
provides  visibility  into  the  effect  of  different  parameters  is  required  (Arora,  Chan,  &  Tiwari, 
2010).  PBL  contracting  is  designed  to  incentivize  this  integration  towards  reducing  life-cycle 
cost  and  improving  design. 

System-level  PBL  contracts  were  developed  to  connect  system  acquisition  and 
logistics  with  a  focus  on  acquiring  a  measurable  performance  outcome  (such  as  the 
availability  of  a  system),  and  they  seek  to  optimize  system  readiness  through  logistics. 
Compared  with  contractor  logistics  support  (CLS),  where  a  contractor,  rather  than  the 
government,  is  responsible  for  the  integration  of  logistics  support  functions,  an  effective  PBL 
requires  a  balanced  contribution  from  both  public-  and  private-sector  providers.  PBL 
contracts,  as  a  group  of  strategies  for  system  support,  are  intended  to  improve  system 
performance  at  a  cost  similar  to  that  previously  achieved  under  a  non-PBL  approach  or 
obtain  the  current  system  performance  at  a  lower  cost.  The  contract  structure  (defining  the 
desired  outcomes),  performance  measurements,  and  pricing  (payment  models)  are  key 
parameters  in  achieving  performance-based  contract  goals  throughout  the  complex  legacy 
system  support  domain.  System-level  PBL  contracts  should  address  the  operational 
availability  time  window,  reliability,  maintainability,  supportability,  operation  and  inventory 
cost,  logistics  footprint,  total  cost  of  ownership,  and  logistics  response  time  for  making 
program  decisions. 

An  alternative  outcome-based  contract  mechanism  called  public-private  partnerships 
(PPPs)  has  been  used  to  fund  and  support  civil  infrastructure  projects.  Availability  payment 
models  for  civil  infrastructure  PPPs  require  the  private  sector  to  take  responsibility  for 
designing,  building,  financing,  operating,  and  maintaining  an  asset.  Under  the  “availability 
payment”  concept,  once  the  asset  is  available  for  use,  the  private  sector  begins  receiving  an 
annual  payment  for  a  contracted  number  of  years  based  on  meeting  performance 
requirements  (Sharma  &  Cui,  2012).  The  challenge  in  PPPs  is  to  determine  a  payment  plan 
(cost  and  timeline)  that  protects  the  public  interest  (i.e.,  does  not  overpay  the  private  sector, 
but  also  minimizes  the  risk  that  the  asset  will  become  unsupported;  Gajurel,  2014). 

Discrete-event  simulation  (DES)  techniques  have  been  previously  used  in  an 
integrated  model  to  optimize  the  payment  and  contract  duration  by  incorporating  the  effects 
of  condition  changes,  uncertainties,  and  required  availability  of  infrastructure  for  PPPs 
(Sharma,  Cui,  Chen  &  Lindly,  2010).  This  work  resulted  in  obtaining  an  improved 
procurement  and  system  acquisition  model  in  which  the  system  availability  was  chosen  as 
the  objective  to  meet  contract  requirements  (Sandborn,  Kashani-Pour,  Zhu,  &  Cui,  2014). 
However,  making  decisions  for  specific  future  actions  during  pre-project  planning  (as  DES, 
which  is  simply  an  implementation  of  discounted  cash  flow  analysis,  does)  does  not 
accurately  address  how  uncertain  conditions  evolve  because  it  does  not  model 
management  flexibility.  Real  options  analysis  (ROA)  is  one  means  of  organizing  and  valuing 
flexible  strategies  to  address  uncertainties  throughout  the  life  cycle  of  systems.  ROA  could 
be  used  to  accommodate  management  flexibility  and  uncertainties  in  both  design  and 
monetary  aspects  of  an  outcome-based  contract. 
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System  Health  Management 

The  maintenance  planning  that  this  paper  focuses  on  is  contingent  on  the  presence 
and  use  of  system  health  management  technologies.  System  health  management 
technologies  such  as  Condition-Based  Maintenance  (CBM)  seek  to  perform  predictive 
maintenance  based  on  the  condition  of  the  system.  Prognostics  and  Health  Management 
(PHM)  uses  the  condition  of  the  system  coupled  with  the  expected  future  environmental 
conditions  (temperature,  vibration,  etc.)  to  forecast  a  Remaining  Useful  Life  (RUL).  The 
system  management  challenge  is  how  to  perform  an  accurate  system  risk  allocation  using 
the  predicted  RULs  (with  their  associated  uncertainties)  to  optimally  plan  when  to  perform 
maintenance  and  allocate  maintenance  resources.  The  optimal  maintenance  planning  is 
modified  by  performance  requirements  imposed  by  the  outcome-based  contracts. 

Maintenance  Planning  Using  Real  Options 

ROA  has  been  previously  applied  to  maintenance  modeling  problems.  An  ROA 
model  for  offshore  platform  life-cycle  cost-benefit  analysis  is  developed  by  treating 
maintenance  and  decommissioning  as  real  options  (Heredia-Zavoni  &  Santa-Cruz,  2004; 
Santa-Cruz  &  Heredia-Zavoni,  2011).  Jin,  Li,  and  Ni  (2009)  presented  an  analytical  ROA 
cost  model  to  schedule  joint  production  and  preventive  maintenance  under  uncertain 
demands.  In  the  study  by  Koide,  Kaito,  and  Abe  (2001),  the  maintenance  and  management 
cost  of  an  existing  bridge  for  30  years  is  analyzed  and  minimized  using  ROA.  Goossens, 
Blokland,  and  Curran  (201 1)  developed  a  model  to  assess  the  differences  in  performance 
between  different  aircraft  maintenance  operations. 

Haddad,  Sandborn,  and  Pecht  (2014)  applied  ROA  to  estimate  the  values  of 
maintenance  options  created  by  the  implementation  of  PHM  in  wind  turbines.  When  an  RUL 
is  predicted  for  a  subsystem,  there  are  multiple  choices  for  the  decision-maker,  including 
performing  predictive  maintenance  at  the  first  maintenance  opportunity,  waiting  until  closer 
to  the  end  of  the  RUL  to  perform  maintenance,  or  doing  nothing  (i.e.,  letting  the  system  run 
to  failure).  Haddad  et  al.  (2014)  demonstrated  that  the  fundamental  tradeoff  in  predictive 
maintenance  problems  with  PHM  is  finding  the  point  in  time  to  perform  predictive 
maintenance  that  minimizes  the  risk  of  expensive  corrective  maintenance  (which  increases 
as  the  RUL  is  used  up),  while  maximizing  the  revenue  earned  during  the  RUL  (which 
increases  as  the  RUL  is  used  up). 

A  Real  Options  Approach  to  Maintenance  Planning  describes  a  real  options 
approach  to  maintenance  planning  when  RULs  are  predicted  for  the  system.  The  section 
titled  Example — Wind  Turbine  With  an  Outcome-Based  Contract  presents  a  case  study  for  a 
PHM  enabled  wind  turbine  with  and  without  an  outcome-based  contract.  In  the 
Generalization  of  Predictive  Maintenance  Options  With  Outcome-Based  Contracts  (Non- 
Production  Earning  Systems)  section,  we  discuss  the  generalization  of  the  approach 
developed  and  demonstrated  in  the  following  two  sections  to  systems  subject  to  other  types 
of  outcome-based  contracts. 

A  Real  Options  Approach  to  Maintenance  Planning 

This  section  starts  with  presenting  the  concept  of  PHM-enabled  maintenance 
options.  Then,  it  describes  how  the  requirements  from  an  outcome-based  contract  are 
incorporated  into  the  option  valuation  process. 

A  real  option  is  the  right,  but  not  the  obligation,  to  undertake  certain  business 
initiatives,  such  as  deferring,  abandoning,  expanding,  staging,  or  contracting.  For  example, 
the  opportunity  to  invest  in  an  asset  is  a  real  “call”  option.  Real  options  differ  from  financial 
options  in  that  they  are  not  typically  traded  as  securities  and  do  not  usually  involve  decisions 
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on  an  underlying  asset  that  is  traded  as  a  financial  security.  Unlike  conventional  net  present 
value  analysis  (discounted  cash  flow  analysis)  and  decision  tree  analysis,  real  options  offer 
the  flexibility  to  alter  the  course  of  action  in  a  real  asset  decision,  depending  on  future 
developments.  Predictive  maintenance  options  are  created  when  in  situ  health  management 
(i.e.,  PHM)  is  added  to  systems.  In  this  case,  the  health  management  approach  generates  a 
remaining  useful  life  (RUL)  estimate  that  can  be  used  to  take  proactive  actions  prior  to  the 
failure  of  a  system.  The  maintenance  option  when  PHM  is  used  is  defined  by  Haddad  et  al. 
(2014) as 

•  Buying  the  option  =  paying  to  add  PHM  to  the  system 

•  Exercising  the  option  =  performing  predictive  maintenance  prior  to  system 
failure  after  an  RUL  indication 

•  Exercise  price  =  predictive  maintenance  cost 

•  Letting  the  option  expire  =  doing  nothing  and  running  the  system  to  failure, 
then  performing  corrective  maintenance 

The  value  from  exercising  the  option  is  the  sum  of  the  cumulative  revenue  loss  and 
the  avoided  corrective  maintenance  cost. 

The  cumulative  revenue  loss  is  what  the  system  would  earn  between  the  predictive 
maintenance  event  and  the  end  of  the  RUL  (if  no  predictive  maintenance  was  done). 
Restated,  this  is  the  portion  of  the  system’s  RUL  that  is  thrown  away  when  predictive 
maintenance  is  done  prior  to  the  end  of  the  RUL.  In  reality,  this  cumulative  revenue  takes 
the  form  of  loss  in  spare  part  inventory  life  (i.e.,  the  revenue  earning  time  for  the  system  will 
be  shorter  because  some  inventory  life  has  been  disposed  of). 

Avoided  corrective  maintenance  cost  includes1  the  avoided  corrective  maintenance 
parts,  service  and  labor  cost,  the  revenue  loss  associated  with  corrective  maintenance 
downtime,  and  the  avoided  under-delivery  penalty  due  to  corrective  maintenance  (if  any). 

Figure  1  illustrates  the  construction  of  the  maintenance  value.  The  cumulative 
revenue2  loss  is  the  largest  on  day  0  (the  day  the  RUL  is  forecasted).  This  is  because  the 
most  remaining  life  in  the  system  is  disposed  of  if  predictive  mainenance  is  performed  the 
day  that  the  RUL  is  predicted.  As  time  advances,  less  RUL  is  thrown  away  (and  less 
revenue  is  lost).  The  avoided  corrective  maintenance  cost  is  assumed  to  be  constant. 


1  This  is  not  the  difference  between  the  predictive  and  corrective  maintenance  actions,  but  rather  the 
cost  of  just  a  corrective  maintenance  event.  The  predictive  maintenance  event  cost  is  subtracted  later 
when  the  real  option  value  is  determined  (i.e.,  in  Equation  1). 

2  The  value  construction  in  this  section  assumes  that  the  system  is  revenue  earning  (e.g.,  a  wind 
turbine  or  an  airplane  used  by  an  airline).  In  Generalization  of  Predictive  Maintenance  Options  With 
Outcome-Based  Contracts  (Non-Production  Earning  Systems),  a  generalization  of  the  model  that 
applies  to  non-production  systems  is  discussed. 
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Figure  1.  Predictive  Maintenance  Value  Construction 

(Lei,  Sandborn,  Goudarzi,  &  Bruck,  2015) 

The  predictive  maintanance  value  is  the  summation  of  the  cumulative  revenue  loss 
and  the  avoided  corrective  maintanance  cost  (Figure  1).  If  there  were  no  uncertainties,  the 
optimum  point  in  time  to  perform  maintenance  would  be  at  the  peak  value  point  (at  the 
RUL),  which  is  the  last  moment  before  the  system  fails.  Unfortunately,  everything  is 
uncertain. 

The  primary  uncertainty  is  in  the  RUL  prediction.  The  RUL  is  uncertain  due  to  inexact 
prediction  capabilities  and  uncertainties  in  the  environmental  stresses  that  drive  the  rate  at 
which  the  RUL  is  used  up.  A  “path”  represents  one  possible  way  that  the  future  could  occur 
starting  at  the  RUL  indication  (Day  0).  The  cumulative  revenue  loss  paths  have  variations 
due  to  uncertainties  in  the  system’s  availability  or  uncertainties  in  how  compensation  is 
received  for  the  system’s  outcome.3  The  avoided  corrective  maintenance  cost  paths 
represent  how  the  RUL  is  used  up  and  vary  due  to  uncertainties  in  the  predicted  RUL.  Each 
path  is  a  single  member  of  a  population  of  paths  representing  a  set  of  possible  ways  the 
future  of  the  system  could  play  out. 

Due  to  the  uncertainties  described  above,  there  are  many  paths  that  the  system  can 
follow  after  an  RUL  indication,  as  shown  in  Figure  2.  Real  options  analysis  lets  us  evaluate 
the  set  of  possible  paths  to  determine  the  optimum  action  to  take. 

Consider  the  case  where  predictive  maintenance  can  only  be  performed  on  specific 

dates.4 


3  For  example,  if  the  system  is  a  wind  turbine,  path  uncertainties  could  be  due  to  variations  in  the 
wind  over  time. 

4  This  could  be  due  to  the  limited  availability  of  maintenance  resources. 
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Figure  2.  Example  of  the  Simulated  Paths  After  an  RUL  Indication 

On  each  possible  maintenance  date,  the  decision-maker  has  the  flexibility  to 
determine  whether  to  implement  the  predictive  maintenance  (exercise  the  option)  or  not  (let 
the  system  run  to  failure  [i.e.,  let  the  option  expire5]).  This  makes  the  option  a  sequence  of 
“European”  options  that  can  only  be  exercised  at  specific  points  in  time  in  the  future.  The  left 
side  of  Figure  3  shows  two  example  predictive  maintenance  paths  (diagonal  lines)  and  the 
predictive  maintenance  cost  (the  cost  of  performing  the  predictive  maintenance).  Real 
Option  Analysis  (ROA)  is  performed  to  valuate  the  option  where  the  predictive  maintenance 
option  value  is  given  by 

OpM  =  Max(VpM  —  CpM,0)  (1) 

where  VPM  is  the  value  of  the  path  (right  most  graph  in  Figure  2  and  the  diagonal  lines  in 
Figure  3)  and  CPM  is  the  predictive  maintenance  cost.  The  values  of  0PM  calculated  for  the 
two  example  paths  shown  on  the  left  side  of  Figure  3  are  shown  on  the  right  side  of  Figure 
3.  Note  that  there  are  only  values  of  0PM  plotted  at  the  maintenance  opportunities  (not  in 
between  the  maintenance  opportunities).  Equation  1  only  produces  a  non-zero  value  if  the 
path  is  above  the  predictive  maintenance  cost  (i.e.,  the  path  is  “in  the  money”). 

Each  separate  maintenance  opportunity  date  is  treated  as  a  European  option.  The 
results  at  each  separate  maintenance  opportunity  are  averaged  to  get  the  expected 
predictive  maintenance  option  value  of  a  European  option  expiring  on  that  date.  This 
process  is  repeated  for  all  maintenance  opportunity  dates.  The  optimum  predictive 
maintenance  date  is  determined  as  the  one  with  the  maximum  expected  option  value.  The 
detailed  mathematical  formulation  of  the  solution  can  be  found  in  Lei  et  al.  (2015). 


5  The  decision-maker  may  also  have  the  flexibility  not  to  implement  the  predictive  maintenance  on  a 
particular  date  but  to  wait  until  the  next  possible  date  to  decide,  which  makes  the  problem  an 
American  option  style  as  has  been  demonstrated  and  solved  by  Haddad  et  al.  (2014).  The  Haddad  et 
al.  (2014)  solution  is  correct  for  the  assumption  that  an  optimal  decision  will  be  made  on  or  before 
some  maximum  waiting  duration  and  the  solution  delivered  is  the  maximum  “wait  to  date.” 
Unfortunately,  in  reality,  maintenance  decision-makers  for  critical  systems  face  a  somewhat  different 
problem:  given  that  the  maintenance  opportunity  calendar  is  known  when  the  RUL  indication  is 
obtained,  on  what  date  should  the  predictive  maintenance  be  done  to  get  the  maximum  option  value. 
This  makes  the  problem  a  European  option  style. 
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Incorporating  Outcome-Based  Contract  Requirements  Into  the  Predictive 
Maintenance  Option 

The  “paths”  described  in  Figures  1  and  2  are  based  on  a  non-outcome-based 
contract  (e.g.,  an  “as-delivered”  energy  delivery  contract  for  a  wind  farm  that  defines  a  single 
fixed  price  for  each  unit  of  the  energy  delivered).  When  a  system  is  managed  via  an 
outcome-based  contract  (like  a  PBL),  the  paths  will  be  impacted.  The  outcome-based 
contract  influences  the  combined  predictive  maintenance  value  paths  due  to  changes  in  the 
cumulative  revenue  loss  and  the  avoided  corrective  maintenance  cost  paths.  These  cost 
paths  will  be  influenced  by  the  outcome  target,  prices  before  and  after  that  target  is  reached 
(generally  the  latter  is  lower  than  the  former),  penalization  mechanisms,  the  outcome 
already  produced,  and  the  operational  state  of  the  other  systems  in  the  population.  For 
example,  assume  that  the  cumulative  outcome  produced  by  a  population  of  systems  is  close 
to  the  outcome  target.  All  systems  are  operational  while  some  are  indicating  RULs.  The 
population  of  systems  can  meet  the  outcome  target  without  the  members  indicating  RULs. 
Then  the  cumulative  revenue  loss  of  the  systems  with  RULs  will  be  lower  than  when  they 
are  managed  under  a  non-outcome-based  contract,  since  the  cumulative  revenue  loss  will 
be  lower  (because  the  price  paid  for  the  outcome  is  lower  after  the  outcome  target  is  met). 
Assume  a  different  scenario  where  the  cumulative  outcome  from  the  population  of  systems 
is  far  from  the  outcome  target  and  many  systems  are  non-operational.  In  this  case,  running 
the  systems  with  RULs  to  failure  and  performing  corrective  maintenance  causing  long 
downtimes  may  result  in  the  population  of  the  systems  not  reaching  the  outcome  target.  In 
this  case,  the  under-delivery  penalty  will  occur,  and  the  avoided  corrective  maintenance  cost 
will  be  higher  than  the  non-outcome-based  contract  (as  delivered)  case  that  doesn’t  have 
any  penalization  mechanisms. 

Under  an  outcome-based  contract,  the  optimum  predictive  maintenance  opportunity 
for  individual  systems  in  a  population  (e.g.,  a  fleet)  is  generally  different  than  for  an 
individual  system  managed  in  isolation.  These  two  cases  would  have  the  same  optimum  if 
an  as-delivered  contract  was  used. 

Example — Wind  Turbine  With  an  Outcome-Based  Contract 

In  this  section,  the  predictive  maintenance  option  model  is  implemented  on  a  single 
turbine,  and  then  a  wind  farm  with  multiple  turbines  is  managed  via  an  outcome-based 
contract.  A  Vestas  V-1 12  3.0  MW  offshore  wind  turbine  (Vestas,  2013)  was  used  for  this 
study. 

Maintaining  offshore  wind  turbines  requires  resources  that  are  not  continuously 
available.  These  resources  include  ships  with  cranes,  helicopters,  and  trained  maintenance 
personnel.  These  resources  are  often  onshore-based  (which  may  be  as  much  as  100  miles 
from  the  wind  farm)  and  may  be  maintaining  more  than  one  wind  farm.  Therefore, 
maintenance  is  only  available  on  scheduled  dates  (maintenance  opportunities)  that  may  be 
weeks  apart.  The  availability  of  maintenance  is  also  dependent  on  weather  and  ocean 
conditions,  making  the  timing  of  future  maintenance  visits  uncertain. 

Figure  4  shows  an  example  result  for  a  single  wind  turbine.  In  this  example,  the  ROA 
approach  is  not  trying  to  avoid  corrective  maintenance,  but  rather  to  maximize  the  predictive 
maintenance  option  value.  In  this  example,  at  the  determined  optimum  maintenance  date, 
the  predictive  maintenance  will  be  implemented  on  only  65.3%  of  the  paths  (the  paths  that 
are  “in  the  money”).  Thirty-two  percent  of  the  paths,  which  are  “out  of  money,”  will  choose 
not  to  implement  predictive  maintenance,  and  in  2.7%  of  the  paths,  the  turbine  has  already 
failed  prior  to  that  date. 
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opportunities 


Figure  3.  Real  Options  Analysis  (ROA)  Valuation  Approach 

Note.  In  the  right  graph,  circles  correspond  to  the  upper  path  and  the  squares  correspond  to 

the  lower  path  in  the  left  graph. 
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Figure  4.  Optimum  Maintenance  Date  After  an  RUL  Indication  for  a  Single  Wind 

Turbine 

The  result  in  Figure  4  assumes  that  all  the  power  generated  by  the  turbine  can  be 
sold  at  a  fixed  price.  There  are  many  wind  farms  (and  other  renewable  energy  power 
production  facilities)  that  are  managed  under  outcome-based  contracts  called  power 
purchase  agreements  (PPAs).  A  PPA  defines  the  energy  delivery  targets,  purchasing  prices, 
output  guarantees,  etc.  Wind  farms  are  typically  managed  via  PPAs  for  several  reasons 
(Bruck,  Goudarzi,  &  Sandborn,  2016).  First,  though  wind  power  can  be  sold  into  the  local 
market,  the  average  local  market  prices  tend  to  be  lower  than  long-term  PPA  contract 
prices.  Second,  lenders  are  not  willing  to  finance  wind  projects  without  a  signed  PPA  that 
secures  a  future  revenue  stream.  Third,  wind  energy  buyers  prefer  simply  purchasing  power 
to  building  and  operating  wind  farms  by  themselves. 

PPA  terms  are  typically  20  years  for  wind  energy,  with  either  a  constant  or  escalating 
contract  price  defined  through  the  whole  term.  At  the  beginning  of  each  year,  a  PPA  often 
requires  the  seller  to  estimate  how  much  energy  they  expect  to  generate  during  the  whole 
year,  based  on  which  an  annual  energy  delivery  target  may  be  defined.  For  each  year,  a 
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maximum  annual  energy  delivery  limit  can  be  set,  beyond  which  a  lower  excess  price  may 
apply.  The  buyer  may  also  have  the  right  not  to  accept  the  excess  amount  of  energy  or 
adjust  the  annual  target  of  the  next  contract  year  downward  based  on  how  much  has  been 
over-delivered.  A  minimum  annual  energy  delivery  limit  or  output  guarantee  may  also  be  set, 
together  with  a  mechanism  to  determine  the  liquidated  damages.  For  example,  the  seller 
must  compensate  the  buyer  for  the  output  shortfall  that  the  buyer  is  contracted  to  receive, 
multiplied  by  the  difference  between  the  replacement  energy  price,  the  price  of  the  energy 
from  sources  other  than  wind  paid  by  the  buyers  to  fulfill  their  demands,  and  the  contract 
price.  The  buyer  may  also  adjust  the  annual  target  of  the  next  contract  year  upward  to 
compensate  for  how  much  has  been  under-delivered. 

Assume  a  5-turb  in  e-farm  managed  via  a  PPA,  turbines  1  and  2  indicate  RULs  on 
Day  0,  turbine  3  operates  normally,  and  turbines  4  and  5  are  non-operational.  Predictive 
maintenance  value  paths  of  all  turbines  with  RULs  need  to  be  combined  together  because 
maintenance  will  be  performed  on  multiple  turbines  on  each  visit  (see  Xin  et  al.  [2015]  for 
details  on  how  the  paths  are  combined  for  multiple  turbines).  Cumulative  revenue  loss, 
avoided  corrective  maintenance  cost,  and  predictive  maintenance  value  paths  for  turbines  1 
and  2  are  shown  in  Figure  5. 


Figure  5.  Combined  Value  Paths  for  Turbines  1  and  2  in  a  5  Turbine  Farm 

Managed  by  a  PPA 

Note.  Some  paths  (as  indicated  by  the  arrow)  change  slopes  because  the  annual  energy 
delivery  target  defined  by  the  PPA  has  been  reached,  after  which  a  lower  price  for  the  power 

applies. 

Real  options  analysis  run  on  the  wind  farm  with  a  PPA  demonstrates  that  the 
maximum  maintenance  value  varies  with  the  number  of  turbines  that  are  down  (non- 
operational).  Figure  6  shows  the  results.  The  result  that  corresponds  to  Figure  5  is  the  right 
most  result  in  Figure  6. 
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Figure  6.  Example  of  the  Simulated  Paths  After  an  RUL  Indication  for  a  Single 
Non-Production  System  Managed  Under  an  Outcome-Based  Contract 

Generalization  of  Predictive  Maintenance  Options  With  Outcome-Based 
Contracts  (Non-Production  Earning  Systems) 

The  real  options  approach  for  the  predictive  maintenance  planning  described  in  the 
sections  A  Real  Options  Approach  to  Maintenance  Planning  and  Example — Wind  Turbine 
With  an  Outcome-Based  Contract  assumes  that  the  system  is  revenue  earning  (e.g.,  a  wind 
turbine  or  aircraft  engine).  In  this  section,  a  generalization  of  the  model  is  developed  and 
applied  to  the  non-production  systems.  For  example,  the  hourly  rate  (e.g.,  per  available 
hour)  in  PBL  contracts  is  a  fixed  number.  Hence,  it  creates  a  different  challenge  than  selling 
the  energy,  which  produces  a  variable  amount  of  revenue. 

To  start  with,  we  assume  a  single  system  (e.g.,  an  aircraft  engine)  with  PHM 
embedded.  This  system  is  managed  under  an  outcome-based  contract  between  a 
contractor  (e.g.,  the  OEM  of  the  engine)  and  a  customer  (e.g.,  an  airline),  in  which  the 
availability  is  the  contracted-for  measurable  performance  outcome.  The  customer  pays  a 
fixed  contract  price  to  the  contractor  for  each  unit  of  time  the  system  is  operating;  the 
contractor  compensates  the  customer  for  each  unit  of  time  the  system  is  down  (non- 
operational).  The  contractor  is  responsible  for  all  the  maintenance  activities.  On  Day  0,  an 
RUL  is  predicted  by  the  PHM,  and  the  contractor  needs  to  decide  if  and  when  to  implement 
the  predictive  maintenance;  alternatively,  the  system  will  be  operated  until  failure,  at  which 
point  corrective  maintenance  will  be  performed  (we  assume  that  safety  is  not  compromised). 
It  is  reasonable  to  assume  the  predictive  maintenance  will  cause  a  lower  cost  (part,  service, 
labor,  etc.)  and  shorter  downtime  than  a  corrective  maintenance. 

Integrated  PHM  and  Inventory  Management 

The  decision  to  act  on  PHM  indications  (RUL  predictions)  will  be  influenced  by  the 
inventory  of  spares  (for  the  system)  that  are  available.  An  integrated  model  to  address  both 
PHM  and  inventory  is  described  here.  This  integration  clarifies  how  PHM  should  be  used  to 
make  maintenance  and  logistics  decisions  and  how  it  impacts  inventory  management.  Here, 
the  primary  focus  is  on  individual  component  prognosis  (e.g.,  an  aircraft  engine  in 
considered  to  be  an  individual  component  for  the  purpose  of  this  discussion)  and  the 
system-level  maintenance  support  and  management  decision. 

Inventory  modeling  is  an  important  part  of  the  integration  of  PHM  and  inventory 
management.  For  example,  Fang  Tu  et  al.  (2007)  have  used  a  multi-state  Markov  network  to 
model  different  levels  of  inventory.  However,  this  model  does  not  consider  the  best  time  to 
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perform  maintenance  (it  only  considers  the  inventory  size).  The  model  discussed  here 
addresses  the  best  time  to  perform  maintenance.  The  goal  of  this  model  is  “when-to-act” 
rather  than  “how  many  spare  parts  to  order.”  This  assumption  allows  this  model  to  be 
extended  to  the  case  of  multiple  systems  using  a  single  shared  inventory  (e.g.,  a  fleet  of 
aircraft  all  drawing  engines  from  the  same  inventory). 

This  model  simulates  the  case  where  upon  RUL  indication,  the  spare  part  is  not 
available  and  it  takes  some  time  ts  for  it  to  become  available.  If  the  maintenance  starts  at  a 
time  point  before  the  spare  part  arrives,  a  penalty  on  the  contractor  will  occur  (e.g.,  to 
expedite  the  spare  order).  In  practice,  ts  is  coming  from  a  probability  distribution  that  models 
the  arrival  of  the  spare  part. 

The  cumulative  revenue  loss,  the  avoided  corrective  maintenance  cost,  and  the 
predictive  maintenance  value  paths  can  be  simulated  as  shown  in  Figure  6.  The  avoided 
corrective  maintenance  cost  in  the  middle  plot  and  the  predictive  maintenance  value  paths 
in  the  right  plot  separate  into  two  groups  where  the  penalty  for  implementing  corrective 
maintenance  before  ts  occurs  to  the  upper  group  of  paths  and  not  in  the  lower. 

By  applying  the  ROA  approach,  the  optimum  predictive  maintenance  date  can  be 
determined,  as  shown  in  Figure  7.  Similar  to  a  wind  farm  managed  under  a  PPA,  when  we 
consider  a  fleet  of  systems  under  an  outcome-based  contract,  both  the  cumulative  revenue 
loss  and  the  avoided  corrective  maintenance  cost  paths  for  the  systems  with  RULs  are 
influenced  by  the  contract  price,  availability  requirement,  penalization  mechanisms,  and  the 
operational  state  of  the  other  systems  in  the  fleet. 


When  fs=0  days,  2  days  or  1 0  days  after  Day  0, 
the  optimum  predictive  maintenance  date  is  3.5 
days  (83  hours)  after  Day  0 


Time  [h] 


Figure  7.  Optimum  Maintenance  Date  Determined  by  the  ROA  Approach  (Pointed 

to  by  the  Arrows) 

Note.  When  ts  changes,  the  optimum  date  may  also  change. 
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Conclusions 

The  objective  of  this  work  is  to  find  the  optimum  predictive  maintenance  opportunity 
for  systems  managed  under  outcome-based  contracts.  Uncertainties  in  the  RUL  predictions 
from  PHM  and  other  sources  are  considered.  This  work  demonstrates  that  the  optimum 
action  to  take  when  a  system  presents  an  RUL  depends  on  whether  the  system  is  an 
individual  or  is  part  of  a  larger  population  of  systems  managed  via  an  outcome-based 
contract. 

When  considering  non-production  systems,  the  availability  of  a  required  spare  part  in 
the  inventory  is  added  to  the  model,  and  both  the  inventory  and  PHM  are  taken  into  account 
when  making  the  decision  on  best  time  to  perform  maintenance. 

Our  vision  is  to  develop  a  multidisciplinary  outcome-based  real  options  pricing  model 
for  supply  chain  and  logistics  design  to  determine  the  optimum  performance  metrics  and  an 
optimum  payment  plan  (amount,  term,  incentive  fees,  and  penalties)  during  the  total  life 
cycle  of  critical  systems  in  PBL  contracts.  The  proposed  integrated  PBL  contract  would 
address  public  policy  and  management  in  the  field  of  government  acquisition  as  well  as 
have  applicability  to  many  types  of  non-governmental  performance-based  contracts.  It 
includes  economics,  financial  management,  risk  management,  marketing,  contracting, 
logistics,  test  and  evaluation,  and  systems  engineering  management. 
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Measuring  the  Return  on  Investment  and  Real  Options  Valuation  of  a  Weather 
Sensor  Bundle  in  Mission  Execution  Processes 

Weather-related  losses  of  remotely  piloted  aircraft  (RPA)  have  exceeded  $100 
million  over  the  past  20  years  (Preisser  &  Stutzreim,  2015).  The  growing  ubiquity  of  RPAs  in 
routine  combat  operations  is  driving  fundamental  changes  to  the  nature  of  support  for  these 
unmanned  aircraft.  Support  requirements  such  as  bandwidth  availability,  data  transmission 
capabilities,  digital  interoperability,  and  weather  forecasting  are  being  pushed  to 
unprecedented  limits  to  ensure  they  enhance  RPA  performance  without  imposing 
superfluous  constraints.  A  persistent  trend  plaguing  RPA  operators  has  been  poor 
environmental  situational  awareness  degrading  overall  operational  effectiveness. 
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The  impact  of  suboptimal  weather  forecasting,  especially  regarding  adverse  weather 
conditions,  on  RPAs  is  significant,  and  it  is  driving  an  increasing  need  for  fundamental 
changes  to  a  system  that  has  matured  over  several  decades  of  proven  operational  success 
with  manned  aircraft.  Without  humans  in  the  cockpit,  the  nature  and  frequency  of  weather 
forecasting  processes  and  supporting  technologies  must  evolve  to  enable  optimized  RPA 
operational  performance  by  providing  weather  products  that  achieve  high  levels  of 
resolution,  accuracy,  and  timeliness. 

This  research  supports  Air  Force  A2I  leadership  by  providing  a  comprehensive 
business  case  analysis  that  estimates  the  overall  value  of  investing  in,  acquiring,  and 
implementing  WeatherNow  technology.  It  provides  a  risk-based  assessment  for  technology 
portfolio  optimization.  The  WeatherNow  technology  in  this  research  refers  to  an  advanced 
weather  forecasting  software  suite  and  an  onboard  weather  sensor.  The  software  suite 
collects,  decodes,  and  processes  space-based,  airborne,  and  surface  observations  used  in 
conjunction  with  numerical  weather  prediction  models.  Using  advanced  algorithms,  data 
fusion  techniques,  and  rapid  update  capability,  it  provides  comprehensive  environmental 
intelligence  products,  improved  asset  protection,  and  decreased  operational  risk.  The 
onboard  weather  sensor  provides  real-time  weather  information  about  icing,  humidity,  and 
cloud  top  heights  directly  to  RPA  aircraft  operators.  The  sensor  also  provides  continuous 
weather  data  in  otherwise  data-deprived  areas.  The  software  suite  and  sensor  were  built  to 
be  integrated  to  provide  timely,  relevant,  and  mission-specific  environmental  intelligence, 
early  threat  detection  for  icing  or  instrument  meteorological  conditions  (IMC),  and  overall 
enhanced  ISR  collection  capability. 

The  study  estimates  the  value  of  WeatherNow  technology  in  terms  of  return  on 
investment  (ROI)  and  uses  integrated  risk  management  (IRM)  to  provide  a  way  to  value 
implementation  options;  both  are  indispensable  tools  that  support  informed  decision-making 
for  technology  investment.  The  analysis  and  conclusions  from  this  study  will  support 
development  of  effective  policy  and  strategic  investment  decisions  in  the  effort  to  transform 
the  existing  weather  forecasting  processes  to  meet  modern  demand  for  near  real-time 
weather  information  to  RPA  operators. 

To  represent  a  typical  mission  execution  process,  this  study  focused  on  an  RQ-4B 
Global  Hawk  squadron  based  at  Beale  Air  Force  Base  (AFB).  The  mission  execution 
process  model  (MEPM)  describes  how  an  RQ-4B  squadron  plans  and  executes  a  typical 
intelligence,  surveillance,  and  reconnaissance  (ISR)  mission.  The  MEPM  consists  of  five 
subprocesses  that  are  further  broken  down  into  tasks.  Each  subprocess  takes  an  input  and 
changes  it  in  some  way  to  produce  an  output,  which  becomes  the  input  for  the  next 
subprocess.  This  process  flow  continues  until  the  final  output  is  produced,  the  RPA  mission 
itself.  The  MEPM  in  this  study  was  verified  by  a  number  of  SMEs  to  be  an  accurate 
representation  while  remaining  generic  enough  to  be  extensible  to  a  wide  range  of  platforms 
and  scenarios  throughout  the  Air  Force  and  the  DoD  at  large.  To  ensure  extensibility  while 
conserving  accuracy  in  the  model,  this  study  is  driven  by  key  assumptions  that  are 
explained  in  further  detail  in  the  study. 

The  quantitative  framework  for  this  research  is  known  as  ROI-IRM  (return  on 
investment  with  integrated  risk  management).  This  methodology  measures  the  value  added 
by  the  WeatherNow  technology  and  by  intangibles  such  as  the  people  executing  the 
process.  Since  traditional  ROI  calculation  is  inadequate  for  assessing  the  value  of  intangible 
assets  such  as  embedded  knowledge,  this  study  uses  the  knowledge  value  added  (KVA) 
methodology  to  estimate  ROI.  The  benefit  of  using  KVA  is  that  a  traditional  metric  such  as 
ROI  can  be  estimated  without  revenue,  by  using  a  surrogate  by  describing  process  outputs 
in  common  units  of  output  (CUO).  Another  benefit  of  KVA  is  its  ability  to  allocate  value 
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across  the  subprocesses  and  even  down  to  the  task  level,  a  much  improved  granularity 
compared  to  traditional  investment  finance  ROI  estimates.  To  measure  the  intangible 
benefits,  KVA  uses  a  metric  called  return  on  knowledge  (ROK).  To  determine  ROI  and  ROK, 
KVA  compares  the  As-ls  MPEM,  the  current  process,  to  the  To-Be  MPEM,  the  process  with 
the  WeatherNow  technology  included.  ROI  and  ROK  estimates  are  precisely  comparable 
with  regard  to  value  for  cost  return  estimates. 

Integrated  risk  management  (IRM)  uses  the  KVA  results  to  further  develop  the 
business  case  by  forecasting  the  future  value  of  technology  options.  IRM  uses  a 
methodology  known  as  real  options  valuation  (ROV)  to  provide  leaders  with  a  robust 
decision  support  tool  to  enable  informed  technology  portfolio  investment  and  implementation 
decisions  based  on  future  value  estimates.  ROI-IRM  is  an  essential  tool  for  supporting 
decisions  on  high  level  strategy  and  policy  concerning  new  technology  and  its  effective 
implementation  and  integration.  KVA  and  IRM  used  together  form  a  powerful  and  defensible 
analytical  tool  set  for  decision-making  for  technology  investments. 

KVA  Analysis  and  Results 

KVA  produces  two  key  metrics,  ROI  and  ROK,  both  expressed  as  ratios.  KVA  takes 
the  traditional  ROI  calculation  used  in  finance  and  adapts  it  to  non-revenue  generating 
organizations  such  as  the  DoD.  As  in  investment  finance,  a  higher  ROI  indicates  a  better 
return  for  the  money  invested.  For  DoD  applications,  a  surrogate  value  for  revenue  must  be 
used  to  monetize  the  outputs  for  purposes  of  an  ROI  estimate  that  typically  comes  from  a 
market  comparable  analysis.  This  research  used  a  very  conservative,  putative  value  of  $1 
per  unit  of  output.  ROK  is  calculated  as  number  of  outputs  (in  common  units)  divided  by  the 
cost  to  produce  the  outputs.  A  higher  ROK  indicates  a  better  use  of  knowledge  assets,  and 
therefore  a  better  investment. 

Overall,  the  results  of  the  KVA  analysis  show  that  the  use  of  WeatherNow 
technology  in  the  RPA  mission  execution  process  will  generate  significantly  higher  returns 
and  far  better  use  of  the  WeatherNow  technology  over  the  current  As-ls  process.  By 
comparing  the  As-ls  MPEM  to  the  To-Be  MPEM,  KVA  not  only  reveals  that  the  WeatherNow 
technology  will  add  value,  but  exposes  which  tasks  benefit  the  most  and  which  benefit  the 
least.  Figure  1  displays  the  differences  in  returns  between  both  models.  With  the 
WeatherNow  technology  included  in  the  process,  ROI  increased  by  69%  and  ROK  is  more 
than  2.8  times  larger  than  the  As-ls  ROK.  These  gains  are  attributable  to  the  large 
improvement  within  the  Flight  Brief/Outbrief/Weather  Update  subprocess,  specifically  the 
Weather  Update  task.  The  WeatherNow  technology  greatly  improves  the  frequency  at  which 
RPA  operators  receive  weather  updates,  from  every  four  hours  in  the  As-ls  process,  to 
every  15  minutes  in  the  To-Be  process.  This  increase  means  an  ROK  almost  300  times 
larger  and  an  ROI  over  1000  times  larger  than  the  As-ls  model.  These  enormous 
improvements  are  due  to  the  process  recognizing  the  added  value  of  the  new  technology 
many  more  times  compared  to  the  As-ls  without  WeatherNow. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-37- 


Figure  1.  Impacts  of  WeatherNow  Technology  Use  on  Mission  Execution 
(Differences  in  Returns  as  Ratios) 

IRM  Analysis  and  Results 

The  IRM  portion  of  this  research  incorporates  raw  data  and  KVA  results  from  a 
concurrent  study  concerned  specifically  with  the  weather  forecasting  process.  Both  studies 
use  the  ROI-IRM  methodologies  and  serve  as  complementary  works.  Three  deployment 
options  were  evaluated  using  IRM  Analysis  of  Alternatives.  The  first  option,  Strategy  A,  is  a 
phased  implementation  in  which  the  WeatherNow  technology  is  implemented  incrementally 
over  time.  The  second  option,  Strategy  B,  is  a  higher  risk  option  in  terms  of  capital 
investment  and  involves  immediate  implementation  and  quick  returns.  The  third  option, 
Strategy  C,  is  to  proceed  with  the  existing  plan  of  implementing  the  new  technology  on  50 
Global  Hawk  aircraft  and  no  more.  Figure  2  displays  the  results  from  the  ROV  analysis. 
Based  on  IRM  economic  valuation  forecasting,  the  highest  value  option  is  to  deploy  the 
WeatherNow  technology  immediately. 
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AS-1S  Strategy 

TO-BE  Strategy:  Sequential  Implementation 

Asset  Value 

S  270.707 

Asset  Value 

S  1  993,256.707 

implementation  Cost 

S  1.342.045 

Implementation  Cost  Phase  1 

S  519.302 

Maturity 

0 

Implementation  Cost:  Phase  li 

S  1.039.605 

Risk-Free  Rate  (Annualized  %) 

0.00% 

Implementation  Cost:  Phase  III 

S  1,639.505 

Dividend  Rate  (Annualized  %) 

0.00% 

Maturity:  Phase  1 

2 

Volatility  (Annualized  %) 

9.85% 

Maturity:  Phase  II 

4 

ROI  % 

-79.83% 

Maturity:  Phase  III 

6 

Net  Present  Value 

S  (1.071.338) 

Risk-Free  Rate  (Annualized  %) 

1.56% 

Option  Value 

S 

Dividend  Rate  (Annualized  %) 

0.00% 

Total  Strategic  Value 

S  (1,071,338) 

Volatility  (Annualized  %) 

30,55% 

Total  Strategic  Value 

S  1,990,841,590 

Incremental  Value  -Added 

S  1,991,912,928 

TO-BE  Strategy:  Immediate  Implementation 

Asset  Value 

S3.9S6.537.4t4 

Real  Options  Valuation 

Implementation  Cost 

S  5. 198.024 

Strategy  A  Phased  Implementation 

5  1.990,641,590 

Maturity 

3 

Strategy  8  Immediate  Execution 

5  3,961,430.393 

Risk-Free  Rate  (Annualized  %) 

092% 

Strategy  C  As-ls  Base  Case 

S  (1,071.333) 

Drndend  Rate  (Annualized  %) 

0  00% 

Volatility  (Annualized  %) 

30.56% 

Total  Strategic  Value 

$3,931,480,893 

Incremental  Value-Added 

$3,982,552,231 

Figure  2.  ROV  Results 

Insights 

Although  enormous  improvements  in  ROI  and  ROK  were  realized,  there  are  still 
more  unrealized  benefits  of  using  WeatherNow  technology.  These  benefits  include  the 
improvement  in  the  richness  of  information  that  RPA  operators  receive  and  the  implications 
of  this  richness  on  the  level  of  confidence  that  operators  have  in  making  critical  go/no-go 
decisions  during  mission  execution. 

Recommendations 

Based  on  the  results  of  this  analysis,  the  following  recommendations  are  submitted. 
To  reduce  uncertainty  and  mitigate  risks,  leaders  should  consider  total  strategic  value 
through  sophisticated  analytical  techniques,  such  as  those  used  in  this  study,  to  inform 
critical  decision-making.  Once  selected,  investments  should  be  tracked  and  monitored  over 
time  and  then  adjusted  as  necessary  based  on  observed  performance.  This  study  was 
designed  around  a  mature  analytical  framework  and  is  extensible  to  a  wide  range  of 
services,  technologies,  and  platforms.  Similar  economic  valuation  analyses  should  be 
performed  on  other  aviation  platforms  that  may  benefit  from  the  WeatherNow  technology, 
particularly  lower  flying  RPA  platforms  that  are  more  limited  by  adverse  weather  than  the 
high-flying  Global  Hawk. 

Conclusion 

This  quantitative  analysis  has  proven  that  implementation  of  WeatherNow 
technology  will  improve  the  current  mission  execution  process  and  has  provided  risk-based 
decision  support  tools  to  assist  with  critical  decisions.  This  research  did  not  examine  the 
socio-technical  implications  of  implementing  such  sophisticated  technology  in  the  mature 
weather  forecasting  system.  Thus,  there  is  opportunity  for  further  research  to  conduct  a 
detailed  examination  of  potential  acceptance  issues  with  WeatherNow  and  how  policy 
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should  evolve  to  support  the  optimal  integration  and  sustained  success  of  WeatherNow 
technology.  This  is  an  important  area  for  continued  research,  investment,  and  innovation, 
toward  modernizing  the  weather  forecasting  system  to  complement  the  unique  needs  of 
RPAs,  improving  their  operational  effectiveness,  and  reducing  their  susceptibility  to  adverse 
weather  conditions. 

Measuring  the  Return  on  Investment  and  Real  Options  Value  of  a  Weather 
Sensor  Bundle  in  Weather  Forecasting  Processes 

Remotely  Piloted  Aircraft  (RPA)  usage  has  grown  exponentially  both  in  ubiquity  and 
utility  over  the  past  decade  and  a  half.  From  their  initial  use  as  a  purely  tactical-level  asset  in 
providing  ground  troops  with  aerial  reconnaissance  and  surveillance,  RPAs  have  become  a 
strategic-level  asset  with  the  precision  strike  capability  to  take  out  high-level  targets 
anywhere  in  the  world.  Currently,  the  greatest  threat  to  RPAs  is  not  surface-to-air  missiles, 
but  rather  their  susceptibility  to  severe  weather  conditions  (Preisser  &  Stutzreim,  2015). 
When  Unmanned  Aerial  Vehicles  (UAVs)  conduct  missions  in  austere  and  remote 
environments  where  little  or  no  infrastructure  exists,  timely  and  accurate  weather  forecasts 
have  become  difficult  and  in  some  cases  almost  impossible  to  produce.  Losses  in  the 
hundreds  of  millions  of  dollars  can  be  attributed  to  UAV  crashes  caused  by  high  winds, 
icing,  lightning,  and  heavy  turbulence  (Preisser  &  Stutzreim,  2015).  Unfortunately  during  the 
development  and  acquisition  of  many  UAVs  in  use  today,  very  little  testing  and  analysis  of 
environmental  situational  awareness  was  conducted  in  order  to  prepare  for  this  threat. 
Furthermore,  without  a  human  present  on  the  platform  itself,  it  becomes  even  more  difficult 
to  determine  current  weather  conditions  throughout  the  mission,  exacerbating  the  threat  that 
severe  weather  creates.  It  is  for  these  reasons  that  a  need  for  increased  weather  situational 
awareness  has  arisen  among  the  UAV  community. 

The  current  weather  forecasting  process  for  UAV  missions  reflects  a  high  degree  of 
uncertainty  and  is  often  based  on  hours-old  and  sometimes  inaccurate  information. 
WeatherNow  technology  will  attempt  to  mitigate  the  risks  presented  by  the  current  weather 
forecasting  process  by  providing  significantly  improved  environmental  awareness  to 
maximize  mission  effectiveness  and  platform  survivability.  The  program  consists  of  an  on¬ 
board  weather  sensor  referred  to  as  an  Atmospheric  Sensing  and  Prediction  System 
(ASAPS)  as  well  as  a  software  suite,  called  Nowcasting,  that  fuses  together  data  from  the 
sensor  as  well  as  from  existing  weather  nodes  (such  as  satellite  imagery  and  ground-based 
radar)  to  create  weather  updates  that  are  accurate,  timely,  and  relevant  to  the  RPA  crew. 

Unique  to  the  WeatherNow  technology  is  the  method  in  which  the  sensor  and 
software  suite  are  able  to  interoperate  and  integrate  with  current  RPA  tactics,  tools,  and 
procedures  (Preisser  &  Stutzreim,  2015).  The  WeatherNow  program  consists  of  three 
separate  phases  that  together  produce  actionable,  real  time,  and  much  improved 
environmental  awareness.  Part  one,  Mission  Area  Sensor  Streaming  (MASS)  retrieves 
environmental  data  from  several  sources,  both  typical  and  atypical  (such  as  overhead 
persistent  infrared)  for  the  area  of  interest.  Part  two,  Dynamic  Rapid  Update  Module 
(DRUM),  fuses  together  the  data  from  the  MASS  phase  (as  well  as  data  retrieved  from  the 
ASAPS  sensor)  to  create  a  4-D  view  of  the  environmental  situation  in  the  targeted  area.  As 
the  name  suggests,  updates  are  conducted  at  a  high  rate,  but  the  system  is  able  to  maintain 
a  low  level  of  latency  while  still  producing  a  high-resolution  view.  The  third  portion  of  the 
Nowcasting  program  is  Fused,  Integrated  Representation  of  the  Environment  (FIRE).  The 
goal  of  FIRE  is  to  provide  the  RPA  crew  with  near-real-time  products  that  give  them 
enhanced  environmental  awareness  of  the  area  of  interest.  The  WeatherNow  program  has 
the  potential  to  significantly  enhance  the  weather  intelligence  gathered  in  support  of 
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unmanned  platform  missions,  but  more  broadly,  it  could  radically  improve  the  weather 
forecasting  process  as  it  exists  in  the  Air  Force  today. 

In  order  to  estimate  the  value  added  by  purchasing  and  implementing  the 
WeatherNow  technology,  it  is  necessary  to  conduct  a  thorough  analysis  of  the  costs  and 
benefits  of  using  both  the  ASAPS  sensor  and  Nowcasting  software  suite.  This  research 
uses  the  Knowledge  Value  Added  (KVA)  methodology  to  quantify  the  benefits  of  introducing 
the  Nowcasting  program  into  the  Air  Force  weather  forecasting  process,  specifically  for  the 
RQ-4B  Global  Hawk  UAV  community.  This  study  quantifies  value  in  terms  of  a  Return  on 
Investment  (ROI),  as  well  as  provides  implementation  options  through  the  use  of  Integrated 
Risk  Management  (IRM)  and  Real  Options  Valuation  (ROV)  portfolio  optimization  strategy. 

This  research  documents  a  process  model  of  the  current  “as-is”  weather  forecasting 
procedures  based  on  input  from  Subject  Matter  Experts  (SMEs)  in  the  9th  Reconnaissance 
Wing  aboard  Beale  Air  Force  Base  (AFB).  The  process  model  describes  how  a  weather 
forecast  is  created  for  use  by  an  RQ-4B  Global  Hawk  squadron  while  remaining  generic 
enough  to  be  applied  to  any  Air  Force  squadron  in  which  weather  forecasts  are  produced. 
The  process  is  broken  down  into  six  main  subprocesses,  which  are  further  disaggregated  to 
capture  the  complex  nature  of  weather  forecasting.  Each  subprocess  takes  a  given  input 
and  produces  an  output,  which  becomes  the  input  to  the  subsequent  subprocess.  The  final 
output  of  the  process  is  an  actionable  weather  forecast  brief  to  be  used  by  the  Global  Hawk 
aircrew. 

KVA  methodology  estimates  the  productivity  embedded  in  an  organization  by 
measuring  the  value  of  knowledge  contained  in  its  people,  technology,  and  processes 
(Housel  &  Bell,  2001 ).  In  this  study,  KVA  quantifies  the  value  of  each  subprocess  of  weather 
forecasting  in  terms  of  a  common  unit  of  output.  In  a  non-profit  organization  like  the  DoD, 
estimating  the  ROI  of  a  technology  investment  in  dollars  is  not  possible  in  the  traditional 
sense.  KVA  produces  a  measure  known  as  Return  on  Knowledge  (ROK)  based  on  the 
knowledge  that  is  embedded  within  the  organization’s  people,  technology,  and  processes. 
This  study  uses  KVA  to  assess  the  value  added  to  the  weather  forecasting  process  by 
implementing  WeatherNow  technology. 

The  IRM  and  ROV  portions  of  this  study  determine  the  different  pathways  for  the 
implementation  of  WeatherNow  into  the  weather  forecasting  process.  Due  to  the  inherent 
volatility  within  the  DoD  acquisition  of  technology,  Air  Force  leadership  needs  to  have  the 
flexibility  to  make  changes  to  their  adoption  strategy.  IRM  and  ROV  provides  those  decision¬ 
makers  with  a  tool  that  helps  optimize  the  value  of  strategic  decisions. 

Knowledge  Value  Added  Results 

As  in  traditional  financial  investment  return  calculations,  ROK  is  determined  by 
dividing  total  output  by  total  input.  In  this  study  the  same  ratio  is  applied  to  calculate  the 
return  on  knowledge  for  each  subprocess  of  weather  forecasting  and  weather  forecasting  as 
a  whole  for  both  the  as-is  model  and  the  to-be  model  (process  with  WeatherNow  technology 
included).  The  numerator  is  calculated  by  multiplying  the  total  learning  time  (time  required  to 
learn  how  to  do  that  specific  task)  by  the  number  of  times  that  task  is  executed  (“fired”)  per 
year,  and  the  value  of  one  hour’s  worth  of  learning  time.  In  this  case  a  value  of  $1 .00  was 
used  as  a  very  conservative  estimate  (this  is  done  in  both  the  as-is  and  to-be  models).  The 
denominator  is  calculated  by  multiplying  the  labor  cost  by  the  number  of  people  performing 
the  task,  the  number  of  times  the  task  is  fired  in  one  year,  and  the  time  required  to  perform 
the  task.  ROK  values  allow  management  to  determine  which  subprocesses  within  their 
organization  add  more  value  to  the  process  as  a  whole.  Ultimately  a  higher  ROK  value  for 
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the  to-be  subprocesses  (as  well  as  the  overall  ROK  value)  would  indicate  that  investing  in 
WeatherNow  technology  adds  value. 

The  results  of  the  KVA  analysis  overwhelmingly  support  the  adoption  of  WeatherNow 
technology  into  the  Air  Force  weather  forecasting  process.  The  mission-watching 
subprocess  received  the  greatest  increase  in  return  on  knowledge  from  the  as-is  to  the  to-be 
scenario,  as  seen  in  Figure  3.  The  reason  for  this  is  because  of  the  increase  in  the  number 
of  times  the  tasks  within  that  sub-process  are  fired  in  one  year.  The  Nowcasting  software 
suite  increases  the  number  of  weather  updates  by  almost  20  times  per  Global  Hawk  flight 
mission.  The  knowledge  embedded  within  the  WeatherNow  technology  is  another  factor  that 
contributes  to  the  increase  in  ROK.  The  Nowcasting  software  and  ASAPS  sensor  take 
thousands  of  hours  of  learning  time  and  are  able  to  fire  at  much  higher  rates  than  humans 
are  capable.  It  is  this  central  principle  that  explains  the  enormous  increases  in  ROK  and 
ROI.  The  return  on  knowledge  in  the  to-be  scenario  is  over  3,000  times  greater  than  the  as- 
is  return  on  knowledge. 


RG-4  Weather  Forecasting  Process: 
Comparison  of  As-ls  and  To-Be 
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Figure  3.  Changes  in  Return  on  Knowledge  and  Return  on  Investment  Due  to 
WeatherNow  Sensor  (Differences  in  Returns  as  Ratios) 

Integrated  Risk  Management  and  Real  Options  Valuation  Analysis  and  Results 

The  IRM  and  ROV  portions  of  this  study  evaluated  three  different  strategies  for 
adopting  the  WeatherNow  Technology.  Strategy  A  implements  both  the  Nowcasting 
software  and  the  ASAPS  sensors  over  time  in  a  phased  approach.  This  is  done  with  the 
intent  to  limit  potential  risks  of  failure  early  in  adoption,  as  technology  and  software 
acquisition  programs  are  prone  to  do.  Phase  I  will  outfit  10  Global  Hawks  with  the  ASAPS 
sensor  within  two  years,  Phase  II  will  outfit  another  20  Global  Hawks  in  the  next  two  years, 
and  Phase  III  will  outfit  another  20  aircraft  within  the  last  two  years.  Strategy  B  is  an 
approach  that  incurs  very  high  capital  investments  early  in  order  to  reap  the  returns  as 
quickly  as  possible.  It  calls  for  the  implementation  of  the  ASAPS  sensor  on  1 00  Global 
Hawks  within  three  years.  Strategy  C  adopts  the  technology  to  only  50  Global  Hawks  to  be 
outfitted  with  the  sensors,  with  no  specific  time  constraint.  The  strategic  option  strategies  are 
seen  in  Figure  4.  As  a  result  of  the  ROV  calculations,  the  most  optimal  solution  is  Strategy 
B,  immediate  execution.  It  produces  a  total  strategic  value  of  just  under  $4  billion,  as 
compared  to  a  negative  strategic  value  of  $1 .07  million  for  the  as-is  strategy.  These  results 
are  seen  in  Figure  5. 
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Figure  4.  Adoption  Strategies  for  WeatherNow 
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Figure  5.  Real  Options  Valuation  Results 

Insights,  Recommendations,  and  Conclusions 

The  KVA  analysis  conducted  in  this  research  indicates  a  favorable  return  should  the 
DoD  decide  to  invest  in  WeatherNow  technology.  Return  on  knowledge  and  cost  savings 
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aside,  WeatherNow  has  potential  benefits  in  several  other  areas  as  well.  This  study  has  only 
looked  at  implementation  on  the  Global  Hawk  platform.  Today  there  are  over  10  different 
RPA  platforms  in  use  by  the  DoD,  all  of  which  are  susceptible  to  adverse  weather 
conditions.  This  study  is  generic  enough  to  be  extensible  to  not  just  Air  Force  weather 
forecasting  in  support  of  Air  Force  only  RPA  platforms.  Army,  Navy,  and  Marine  Corps 
forces  are  potential  benefactors  of  WeatherNow  technology  as  well.  Furthermore,  the 
accurate  weather  forecasts  produced  by  the  Nowcasting  software  suite  are  not  necessarily 
for  use  by  RPA  aircrews  only.  Manned  aircraft  have  the  potential  to  benefit  from  the 
increased  environmental  awareness  afforded  by  WeatherNow.  Additionally,  ground  units, 
specifically  those  that  fire  long-range  rockets  like  the  High  Mobility  Artillery  Rocket  System 
(HIMARS)  and  Guided  Multiple  Launch  Rocket  System  (GMLRS)  rely  on  timely  and 
accurate  weather  forecasts.  Improved  weather  intelligence  would  help  those  units  improve 
the  accuracy  and  lethality  of  their  strike  missions.  As  with  most  technological  innovations 
that  may  disrupt  current  practices,  however,  appropriate  care  and  time  must  be  taken  to 
train  personnel  in  the  operations  and  implications  of  WeatherNow  technology.  The  relevant 
publications  and  doctrine  would  also  have  to  reflect  the  use  of  WeatherNow  as  well.  It  is  the 
recommendation  of  this  study,  however,  that  Air  Force  leadership  adopts  this  technology 
and  implements  it  rapidly. 

Conclusion 

This  quantitative  analysis  supports  the  conclusion  that  implementation  of  the 
WeatherNow  technology  that  was  examined  for  this  study  will  improve  the  current  mission 
execution  process  and  real  time  weather  forecasting  process.  The  results  also  have 
provided  a  risk-based  decision  support  framework  and  supporting  tool  set  to  assist  with 
future  investment  in  technology  decisions  by  treating  such  decisions  as  a  portfolio  of  options 
with  varying  future  quantitative  values  and  risks. 

The  focus  of  this  research  precluded  examining  the  socio-technical  implications  of 
implementing  such  sophisticated  weather  forecasting  technology  in  the  current  weather 
forecasting  system.  Thus,  there  is  opportunity  for  further  research  to  conduct  a  detailed 
examination  of  potential  acceptance  issues  with  WeatherNow  and  how  policy  should  evolve 
to  support  the  optimal  integration  and  sustained  success  of  WeatherNow  technology.  This  is 
an  important  area  for  continued  research,  investment,  and  innovation,  all  in  the  course  of 
modernizing  the  weather  forecasting  system  to  complement  the  unique  needs  of  RPA  pilots. 
By  improving  their  operational  effectiveness  and  reducing  their  susceptibility  to  adverse 
weather  conditions,  the  number  of  successful  missions  will  increase  over  time. 

Recommendations 

The  results  clearly  indicate  that  the  immediate  option  to  deploy  the  WeatherNow 
technology  RAP  fleet-wide  are  warranted.  Delays  in  acquiring  and  implementing  this 
technology  will  likely  result  in  reduced  value  added  and  lower  than  possible  mission 
success.  The  effect  of  this  technology  on  mission  success  should  be  tracked  over  time  so 
that  options,  risks,  and  ROIs  can  be  adjusted  to  reflect  real  usage  of  the  technology. 

The  performance  analytical  framework  used  in  this  study  is  extensible  to  a  wide 
range  of  services,  technologies,  and  platforms  beyond  its  use  in  evaluating  the  potential 
value  added  of  the  WeatherNow  technology.  Similar  economic  valuation  analyses  should  be 
performed  on  other  aviation  platforms  that  may  benefit  from  the  WeatherNow  technology, 
particularly  lower  flying  RPA  platforms  that  are  more  limited  by  adverse  weather  than  the 
high-flying  Global  Hawk. 
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Abstract 

The  systems  engineering  process  to  design  and  develop  new  systems  is  based  on  a 
technical  rationalization  of  the  design  process.  This  paper  contrasts  the  technical  rational 
approach  with  the  design  thinking  approach,  which  describes  the  principles  and  methods 
based  on  how  experienced  designers  approach  design  problems.  We  assert  the  structure  of 
the  design  problem  changes  during  development,  and  one  contributor  to  the  challenges  that 
defense  programs  face  in  meeting  budget,  schedule,  and  performance  requirements  is  the 
mismatch  between  the  nature  of  the  design  problem  and  the  engineering  approach.  Our 
position  is  a  variant  of  contingency  theory,  contending  there  is  no  single  best  way  to 
approach  a  problem,  and  an  approach  effective  in  one  situation  may  not  be  effective  in 
another.  This  paper  reviews  the  technical  rational  and  design  thinking  perspectives.  The 
paper  then  examines  the  systems  engineering  process  in  light  of  design  thinking  principles 
and  methods,  and  the  paper  makes  recommendations  to  partition  development  into 
architecting  and  engineering,  increase  the  variety  and  frequency  of  prototyping,  explicitly 
show  iteration  in  process  models,  and  practice  delayed  commitment. 

Introduction 

The  defense  acquisition  system  implements  systems  engineering  through  standards, 
codification  of  policies  and  procedures,  and  extensive  documentation.  The  systems 
engineering  vee  is  the  process  model  and  serves  as  the  de  facto  standard  process  model 
for  Department  of  Defense  (DoD)  programs.  The  vee  process  model  is  a  top-down  approach 
of  analyzing  stakeholder  needs  to  arrive  at  technical  system  requirements  and  finally  a 
system  design.  The  top-down  approach  is  evidence  in  the  extensive  decomposition  from  the 
system-level  design  to  subsystem  design  and  component  design.  The  vee  model  then 
shows  synthesis  by  building  and  integrating  the  system  from  a  bottom-up  perspective.  This 
is  followed  by  component  level,  subsystem  level,  and  finally  system-level  test  and 
evaluation.  The  vee  model  makes  feedback  explicit  in  verification  and  validation  information 
flows  from  test  and  evaluation  to  the  analysis  and  design  activities. 
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The  systems  engineering  vee  model  adheres  to  the  technical  rational  perspective.  In 
this  paper,  we  review  the  technical  rational  design  approach  and  the  assumptions 
underlying  its  methods.  We  then  introduce  design  thinking  and  its  assumptions.  The 
technical  rational  design  approach  and  the  design  thinking  approach  start  with  different 
worldviews  and  lead  to  two  very  different  design  approaches.  We  then  analyze  the  systems 
engineering  process  in  order  to  make  recommendations  to  improve  the  process.  We  make 
recommendations  and  draw  final  conclusions. 

Technical  Rational  Design 

The  Technical  Rational  Design  approach  is  a  structured  approach  to  design  based 
on  a  problem-solving  perspective  in  which  the  designer’s  task  is  to  solve  a  design  problem. 
Simon  (1996)  was  among  the  first  to  present  the  problem-solving  perspective  of  design, 
which  separates  design  into  a  problem  formulation  phase  and  problem  solution  phase. 
Simon  and  the  artificial  research  community  at  the  time  sought  computer  algorithms  to  do 
the  design  process.  The  technical  rational  design  approach  assumes  a  positivist  perspective 
that  a  single  objective  truth  exists  and  can  be  observed  and  discovered  through  scientific 
methods  (Neuman,  2005). 

Pahl  and  Beitz  (2013)  wrote  an  influential  German  text  defining  a  systematic 
approach  to  engineering  design,  which  illustrates  the  assumptions  and  perspective  of 
technical  rational  design.  They  partition  the  design  process  into  four  phases  of  clarifying  the 
task,  conceptual  design,  embodiment  design,  and  detail  design.  The  design  process  starts 
with  the  definition  of  requirements  followed  by  successful  refinement  of  a  design  concept 
through  the  last  three  phases.  Each  step  of  the  way,  the  designer  is  making  rational 
decisions  in  a  pre-determined  manner  to  arrive  at  the  final  design. 

The  technical  rational  design  approach  makes  two  key  and  interrelated  assumptions. 
First,  technical  rational  design  approach  assumes  problem  formulation  can  be  separated 
from  problem  solution.  We  see  evidence  of  this  mindset  in  many  texts  with  the  advice  to 
separate  the  “what”  described  by  the  functional  architecture  from  the  “how”  described  by  the 
physical  architecture  (see  Blanchard  &  Fabrycky,  1990).  Second,  the  technical  rational 
design  approach  assumes  we  can  know  and  present  the  stakeholder  objectives  and  system 
requirements  without  embarking  on  any  design  activities.  The  designer  would  then  be  able 
to  search  the  design  space  to  determine  the  set  of  Pareto  optimal  designs. 

Given  these  two  assumptions,  design  can  progress  in  an  orderly  fashion  through 
each  step  with  minimal  feedback  and  iteration.  Moreover,  adopting  these  assumptions 
makes  the  design  problem  amenable  to  formulation  as  a  mathematical  problem,  which  can 
then  be  subjected  to  algorithms  to  find  the  best  designs.  Here  we  formulate  the  design 
problem. 

Design  variables  are  the  controllable  dimensions,  characteristics,  and  attributes  of  a 
system  design  specification.  Initially,  the  value  for  each  design  variable  is  unknown,  and 
through  the  process  of  design,  the  designer  will  specify  values  for  the  design  variable  until 
all  design  variables  are  specified.  Let  dj  denote  the  ith  design  variable  which  can  take  any 
value  in  A. ,  in  other  words  dt  e  A(. .  The  set  A.  can  be  the  set  of  integers,  real  numbers,  or 
discrete  options  available  for  that  design  parameter  (e.g.,  if  dj  is  the  design  parameter  for 
battery  type,  then  the  domain  A.  =  {lithium-ion,  nickel-cadmium,  lead-acid}).  If  there  are  n 

design  parameters,  then  the  design  space  is  an  n-dimensional  hyperspace  that  contains  all 
the  possible  designs.  It  is  defined  by  the  Cartesian  product 
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DS  =  A,  x  A,  x...x  A 

12  n 

A  design  denoted  by  Dk  is  a  vector  of  length  n  that  specifies  a  value  for  each  of  the 
design  variables,  i.e.,  Dk  =  (dk  ,...,dk) .  The  superscript  denotes  the  kth  design  and 

distinguishes  between  the  many  designs  in  a  design  space.  Every  point  in  the  design  space 
is  a  design.  However,  not  every  design  in  DS  will  satisfy  stakeholder  requirements  or  even 
be  technically  feasible. 

Requirements  either  describe  function  relationships  between  multiple  design 
variables  or  requirements  place  restrictions  on  the  admissible  values  of  a  design  variable. 
The  jth  function  requirement  is  given  by 

Vj  (Dk  )  <  0  j  =  \...m 

and  requirement  restrictions  are  expressed  by  lower  limits  Af  and  upper  limits  Af  on  the 
admission  values  as  Af  <  di  <  A f . 

The  jth  system  requirement  partitions  the  design  space  into  a  region  that  satisfies 
the  requirement,  DS*  and  a  region  that  does  not  satisfy  the  requirement,  DS* .  A  design 

Dk  satisfies  a  system  requirement  if  it  is  in  the  satisfactory  region  of  the  requirement 
defined  by  DS*  c=  DS  .  The  intersection  of  all  m  requirements  defines  the  satisfactory 
region  within  which  each  design  satisfies  all  the  system  requirements,  and  is  given  by 

m 

DS*  -  P|  DS*  . 

r= i 

A  design  team  will  seek  the  best  design,  in  other  words  the  design  that  delivers  the 
most  value  to  the  stakeholders,  within  the  satisfactory  region.  Almost  all  designs  will  have 
multiple  objectives  from  which  stakeholders  derive  value.  The  value  of  a  design  with  respect 
to  a  single  objective  is  given  by  a  value  function.  Value  is  a  function  of  the  design 
parameters  and  noise  parameters.  The  value  of  the  kth  design  with  respect  to  the  Ith 
objective  is  given  by  the  value  function 

The  set  of  noise  parameters,  denoted  by  n1(...,np,  represents  uncontrollable 
influences  on  performance  such  as  environmental  factors. 

The  vector  Vk  ={vk  denotes  the  values  of  the  kth  design  across  all 

objectives.  A  design  with  a  value  of  \a  =  )  is  said  to  dominate  a  design  with  a 

value  of  V*  =  (vib,...,Vnb'j  if  and  only  if  Va  is  partially  less  than  V*,  which  is  when 

\/l  e.LV,a  >Vlb  a31  e  L,V{a  >Vlb . 

The  set  of  dominate  designs  is  called  the  Pareto  frontier.  We  speak  of  designers 
trading  off  objectives,  and  they  would  do  this  between  designs  in  the  Pareto  frontier. 
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In  summary,  the  design  problem  is  formulated  as  finding  the  design(s)  that  maximize 
value  while  satisfying  all  the  system  requirements.  It  is  expressed  by  the  optimization  model 


argmax  Vk  =(vxk  ,...,Vkn) 

nk 


r.  (Dk  )  <  0  j  =  \...m 

Af  <  dt  <  A"'  i  =  \...n 


The  design  optimization  model  is  possible  in  technical  rational  design  because  the 
problem  structure  is  assumed  to  be  well-defined,  it  is  assumed  we  can  express  value 
mathematically,  and  it  is  assumed  we  can  express  all  requirements  as  mathematical 
functions.  The  design  problem  then  becomes  a  matter  of  searching  the  design  space  to  find 
the  Pareto  optimal  designs. 

The  concepts  and  assumptions  of  the  technical  rational  design  approach  form  the 
basis  upon  which  systems  engineering  process  models  (Blanchard  &  Fabrycky,  1990)  and 
the  majority  of  engineering  design  education  (Dym  et  al.,  2005).  The  waterfall  model  was  an 
early  example,  largely  developed  in  reaction  to  the  poor  experience  of  development 
software  without  any  process. 

There  are  many  benefits  to  the  technical  rational  design  approach  embodied  by 
these  methods.  The  systemization  of  design  leads  to  manageable  projects  and  the  ability  to 
define  milestones  and  deliverables,  and  it  standardizes  the  process  which  facilitates 
communication  and  makes  the  process  repeatable.  These  benefits  have  enhanced 
government’s  and  industry’s  ability  to  design  and  develop  complex  weapon  systems. 

Design  Thinking 

Design  thinking  is  a  term  to  describe  the  creative  thinking  process  exhibited  by 
designers  and  now  used  in  many  non-traditional  design  domains  such  as  strategy 
formulation,  business,  and  social  sciences  (Brown,  2008;  Plattner  et  al.,  2010).  It  has  also 
influenced  the  Navy,  as  seen  in  ADM  Richardson’s  eight-page  “A  Design  for  Maintaining 
Maritime  Superiority”  strategic  document. 

Dorst  (2010)  differentiates  design  thinking  from  other  thought  processes  through  the 
logical  process  of  abduction,  whereby  we  know  the  end  value  we  want  and  have  to  discover 
the  means  to  achieve  it.  The  study  and  conceptualization  of  design  thinking  is  conducted 
primarily  according  to  an  interpretivism  approach  after  Schon’s  (1983)  reflection-in-action 
research,  in  which  he  examined  how  professionals  actually  work.  Interpretivism  accepts 
multiple  different  realities  based  on  the  observer’s  perspective.  It  is  in  contrast  to  the 
positivist’s  claim  that  there  is  a  single  objective  reality  and  we  can  only  acquire  knowledge 
through  the  scientific  method,  which  is  the  technical  rational  approach  (Neuman,  2005). 

A  process  for  design  thinking  identifies  five  activities  (after  Stanford  University 
Institute  of  Design,  2016): 


1 .  Empathize — Understand  what  the  stakeholders  desire  through  open-ended 
questions  and  related  techniques  to  better  understand  the  problem  from 
many  different  perspectives. 

2.  Define — Combine  and  synthesize  all  the  acquired  information  and 
perspectives  to  arrive  at  a  group  consensus  on  the  problem  structure. 

3.  Ideate — Generate  ideas  in  a  typical  brainstorming  fashion  with  the  goal  to 
generate  as  many  ideas  as  possible. 
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4.  Prototype — Create  a  mock-up  of  the  design  solution  and  use  it  for  evaluation. 

5.  Test — Test  the  prototype,  preferably  with  stakeholders  and  end-users. 

Completion  of  a  single  iteration  leads  to  a  greater  understanding  of  the  problem  as 
well  as  a  potential  design  solution.  Design  thinking  is  based  on  the  observation  that 
designers  work  simultaneously  on  both  problem  structuring  and  problem  solving  (Dorst  & 
Cross,  2001).  Problem  structuring  involves  the  discovery  of  needs,  requirements,  and 
feasibility  so  that  the  designer  can  understand  the  problem.  Problem  structuring  is  achieved 
partially  by  proposing  solutions  because  having  a  solution  provides  something  concrete  for 
stakeholders  to  react  to  and  better  understand  their  needs. 

Design  thinking  is  also  referred  to  as  human-centric  design  because  of  the 
importance  placed  on  empathizing  with  the  human  users  (Patnaik,  2009).  During  the 
empathize  step,  designers  frame  and  re-frame  the  problem  by  adopting  the  user’s 
perspective  to  arrive  at  different  problem  structures.  Framing  the  problem  from  multiple 
perspectives  implies  the  imposition  of  an  interpretation  of  the  problem,  and  each 
interpretation  allows  for  additional  insights  and  potentially  different  and  more  fruitful 
solutions  (Paton  &  Dorst,  2011). 

Unlike  technical  rational  design,  design  thinking  seeks  to  preserve  ambiguity  as  long 
as  possible  because  too  quickly  converging  on  a  solution  is  seen  to  stifle  creativity.  Design 
thinking  also  promotes  the  early  and  frequent  creation  of  prototypes  to  serve  multiple 
purposes  from  problem  understanding,  solution  evaluation,  and  communication. 

Analysis  and  Recommendations 

This  section  is  organized  according  to  the  main  recommendations  on  how  design 
thinking  can  be  incorporated  into  the  systems  engineering  process. 

Architecting  vs.  Engineering 

The  design  problem  changes  in  character  from  an  ill-structured  problem  in  the  early 
phases  to  a  well-structured  problem  in  the  later  phases.  Consequently,  it  makes  sense  to 
approach  the  different  design  problems  differently.  The  concept  of  tailoring  is  based  on 
contingency  theory,  which  claims  the  best  approach  depends  on  the  fit  between  the  process 
and  contextual  factors  (Drazin  &  Van  de  Ven,  1985).  In  the  systems  engineering  process,  a 
major  contextual  factor  is  the  nature  of  the  design  problem:  ill-structured  or  well-structured. 

DoD  Instruction  (DoDI)  5000.02  ( Operation  of  the  Defense  Acquisition  System) 
allows  for  tailoring  and  says,  “The  structure  of  a  DoD  acquisition  program  and  the 
procedures  used  should  be  tailored  as  much  as  possible  to  the  characteristics  of  the  product 
being  acquired,  and  to  the  totality  of  circumstances  associated  with  the  program  including 
operational  urgency  and  risk  factors.”  The  instruction  provides  four  baseline  acquisition 
models  to  serve  as  starting  points  for  tailoring.  What  is  lacking  in  the  systems  engineering 
community  is  guidance  on  how  to  make  the  tailoring  decisions. 

The  design  process  should  be  partitioned  between  two  distinct  phases  of 
architecture  design  and  system  design.  The  architecture  phase  should  be  managed 
according  to  a  design  thinking  approach,  and  the  system  design  phase  according  to  the 
technical  rational  design  approach.  Architecting  is  the  activity  comprising  the  generation, 
evaluation,  and  selection  of  alternative  solutions.  The  architect  works  in  both  the  problem 
space  and  the  design  space.  Understanding  the  problem  and  conceiving  of  a  design 
solution  are  directly  related  to  each  other.  Consequently,  architects  iterate  between  problem 
structuring  and  problem  solving  and  in  the  process  they  reveal  new  understandings  of  the 
problem  space  and  the  solution  space. 
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The  output  of  the  architecture  design  phase  is  a  system  architecture  defining  the 
structure  of  the  system  in  terms  of  the  design  variables,  set  of  system  technical 
requirements,  and  the  measures  of  effectiveness,  which  in  the  DoD  define  value. 
Consequently,  we  have  a  well-defined  problem  amenable  to  the  technical  rational  design 
approach.  Designers  would  search  the  design  space  using  algorithms  and  computational 
tools  when  available  and  appropriate  to  find  the  set  of  Pareto  optimal  design  solutions. 

We  note  the  systems  engineering  community  has  been  moving  to  this  dichotomy 
between  system  architecting  and  system  engineering,  as  evidenced  by  the  earliest  book  on 
system  architecture  (Rechtin  &  Maier,  2010),  to  more  recent  works  and  emphasis 
(Dickerson  &  Mavris,  2009). 

Requirements 

Both  technical  rational  design  and  design  thinking  suggests  we  need  to  think  of 
systems  requirements  as  being  of  two  types:  value  statements  and  technical  system 
requirements.  Value  statements  express  what  stakeholders  value  in  a  system,  can  be 
measured  on  a  continuous  scale,  and  are  negotiable.  Requirements  are  the  constraints  a 
system  must  have  and  are  non-negotiable.  In  the  design  optimization  model,  the  value 
statements  are  part  of  the  objective  function  and  the  requirements  define  the  edges  of  the 
design  space.  When  we  state  stakeholder  value  as  a  requirement  rather  than  a  value 
statement,  we  shackle  the  hands  of  our  designers  by  unnecessarily  restricting  the  design 
space.  The  value  statements  more  closely  match  attainment  of  value  as  defined  by 
stakeholders.  Barry  Boehm  came  to  a  similar  conclusion  and  suggested  we  need  to  modify 
our  terminology  in  order  to  effect  the  cultural  change  within  the  acquisition  and  systems 
engineering  communities  (Mavor  &  Pew,  2007). 

Since  the  set  of  requirements  define  the  edges  of  the  design  space,  it  is  easily 
shown  that  adding  requirements  makes  the  design  space  smaller  or  at  best  the  same  size.  If 
the  design  space  is  made  smaller,  then  it  is  possible  good  designs  are  excluded.  Given  this 
insight,  it  is  important  to  keep  to  a  minimum  the  number  of  technical  system  requirements 
because  they  limit,  perhaps  unnecessarily  in  some  cases,  the  design  space. 

Prototyping 

Prototyping  during  the  early  architecting  phase  is  as  important  as  during  the  later 
phases  (Kimbell,  201 1 ).  It  seems  many  programs  illogically  think  a  prototype  is  an  almost 
fully-functional  copy  of  the  intended  system.  Prototyping  in  the  design  thinking  community  is 
much  more  inclusive.  Prototyping  during  the  architecting  phase  is  important  for  reasons  of 
discovery,  developing  a  deeper  understanding  of  stakeholder  value,  communication,  and  to 
support  problem  structuring.  A  prototype  as  discussed  by  the  design  thinking  community  is 
any  physical  model  that  stakeholders  and  the  designers  can  interact  with.  Design  thinking 
promotes  the  building  and  usage  of  many  low  fidelity  prototypes  to  aid  the  designers  during 
problem  structuring.  An  overemphasis  by  many  programs  on  high  fidelity  prototypes  with 
much  of  the  functionality  of  the  expected  production  system  is  counterproductive  because 
they  overlook  the  value  of  prototyping  in  the  early  architecting  phase.  Programs  need  to 
expand  their  prototyping  capability  in  terms  of  both  the  diversity  and  fidelity  of  prototypes. 

Incremental  and  Iterative 

Design  thinking  research  has  demonstrably  revealed  that  higher  performing 
designers  iterate  between  problem  structuring  and  problem  solving  (Dorst  &  Cross,  2011). 
Top-down,  sequential  process  models  such  as  the  vee  model  do  not  show  this  important 
aspect  of  system  design  and  development.  Moreover,  the  systems  engineering  vee  and  the 
Joint  Capability  Integrated  Development  (JCIDS)  process  suggest  it  is  possible  for  the 
government  to  generate  a  solution  agnostic  specification  of  capability  needs  and  system 
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requirements.  Design  thinking  says  such  a  separation  is  not  possible.  In  fact,  designers 
need  to  think  about  solutions  in  order  to  better  understand  needs  and  system  requirements. 
The  systems  engineering  models  should  incorporate  documentation  to  stress  the 
importance  of  both  incremental  and  iterative  development.  Larman  and  Basili  (2003)  discuss 
the  history  of  incremental  and  iterative  development  and  why  within  the  software  domain 
these  methods  are  usually  superior  to  sequential  and  document-intensive  methods. 

The  number  of  iterations  in  iterative  approaches  is  limited  by  either  time  or  budget. 
Consequently,  it  is  impossible  to  exhaustively  search  the  entire  design  space  before  running 
out  of  time  or  money.  All  iterative  approaches  are  local  searches  confined  by  the  starting 
point  and  consequently,  if  you  have  a  poor  starting  point,  you  will  likely  finish  at  an  inferior 
design.  One  strategy  is  the  multi-start  whereby  instead  of  using  a  single  starting  design  to 
iterate  upon,  the  designers  consider  multiple  alternative  designs  preferably  representative  of 
the  entire  design  space.  Indeed,  a  GAO  (2009)  report  analyzed  32  major  defense  programs 
that  started  after  the  year  2003.  The  GAO  found  the  programs  with  a  broad  scope  of 
alternatives  had  lower  cost  and  schedule  growth  than  programs  with  a  narrow  scope  of 
alternatives.  Each  alternative  is  essentially  a  starting  design  for  a  multi-start  strategy  to 
explore  the  design  space.  A  broader  AoA  is  more  likely  to  fully  explore  the  design  space  and 
lead  to  better  program  outcomes.  A  narrow  AoA  is  less  likely  to  fully  explore  the  design 
space;  hence  the  problems. 

Deferment  and  Delayed  Commitment 

The  architecting  phase  is  characterized  by  high  uncertainty,  yet  it  is  well  established 
that  early  design  decisions  can  have  an  enormous  impact  on  committed  cost  (Blanchard  & 
Fabrycky,  1990).  Deferring  decisions  until  more  information  can  be  gained  is  a  good 
strategy  (Loch  &  Terwiesch,  2005).  Set-based  design,  based  upon  American  understanding 
of  Toyota’s  design  process,  is  when  instead  of  iterating  from  a  starting  design,  a  set  of 
designs  is  propagated  and  progressively  pruned  until  a  final  design  is  found  (Sobek  et  al., 
1999).  Set-based  design  is  one  approach  to  tackling  the  mismatch  between  the  amount  of 
information  available  and  the  timing  of  decisions.  It  delays  decisions  until  more  information 
is  available.  This  is  a  form  of  progression  refinement  since  as  the  development  process 
progresses,  the  uncertainty  (measured  as  the  size  of  the  set)  is  gradually  decreased  until  a 
precise  value  is  arrived  at.  Giachetti  et  al.  (1999)  did  something  similar  with  fuzzy  sets;  Finch 
and  Ward  (1995)  with  intervals;  HP  with  delayed  differentiation;  and  Boehm  and  Lane 
(2007)  with  delayed  commitment.  More  recently,  the  set-based  approach  has  been  applied 
to  naval  ship  design  (Singer  et  al.,  2009;  Mebane  et  al.,  201 1 ). 

Conclusions 

Design  thinking  starts  out  with  a  very  different  worldview  from  the  technical  rational 
design  approach.  While  technical  rational  design  is  based  on  a  positivist  perspective  of 
knowledge,  design  thinking  is  based  on  an  interpretative  perspective.  The  result  is  very 
different  assumptions  about  how  to  conduct  design,  and  consequently  very  different 
approaches.  Using  contingency  theory,  we  propose  to  partition  the  system  design  and 
development  process  to  achieve  a  better  match  between  the  problem  space  and  the 
solution  approach.  Broadly,  this  means  separating  design  and  development  into  two  phases 
of  architecting  and  engineering  design.  The  architecting  phase  is  guided  primarily  by  the 
design  thinking  perspective,  and  the  engineering  design  phase  is  guided  primarily  by  the 
technical  rational  design  perspective.  Additionally,  we  make  recommendations  for  adoption 
of  a  broader  set  of  prototyping  capabilities,  rethinking  many  requirements  as  value 
statements,  and  for  greater  recognition  of  iteration  and  incremental  development  in  the 
systems  engineering  process  model.  The  Systems  Engineering  Department  at  the  Naval 
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Postgraduate  School  (NPS)  is  working  towards  educating  the  younger  cohort  of  naval 

engineers  in  design  thinking  and  how  it  can  be  beneficially  incorporated  into  the  systems 

engineering  process. 
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Abstract 

This  paper  examines  the  role  of  content  analysis  in  systems  engineering  technical  evaluation 
processes.  Content  analysis  is  a  qualitative  data  analysis  methodology  used  to  discover 
consistencies,  inconsistencies,  themes,  and  trends  within  datasets.  This  methodology  is 
particularly  useful  when  evaluating  Contract  Data  Requirements  List  documents,  as  well  as 
deficiency  reports  from  test  and  evaluation  activities;  examples  of  such  analyses  are 
provided.  Factors  that  can  impact  a  systems  engineer’s  ability  to  effectively  and  efficiently 
use  this  analysis  method  are  also  discussed.  Research  into  the  development  of  valid, 
relevant,  and  repeatable  analysis  criteria  promises  to  define  (1)  how  content  analysis  can  be 
used  consistently  across  different  system  baselines  and  (2)  how  content  analysis  results 
generated  during  the  “Production  and  Deployment”  and  the  “Operations  and  Support” 
acquisition  lifecycle  phases  can  be  used  to  shape  requirements  definitions  for  system 
upgrade  or  modification  contracts  and  new  baseline  contracts.  Finally,  content  analysis 
training  and  skill  development  for  systems  engineers  in  the  acquisition  workforce  is 
discussed. 

Introduction 

During  the  different  phases  of  a  system’s  lifecycle,  systems  engineers  evaluate  a  lot 
of  data  from  a  variety  of  sources.  A  key  part  of  analyzing  this  data  is  discovering  patterns 
and  using  those  patterns  to  support  additional  analyses.  As  stated  in  the  International 
Council  on  Systems  Engineering  (INCOSE)  Handbook, 

Systems  thinking  captures  and  exploits  what  is  common  in  a  set  of  problems 
and  corresponding  solutions  in  the  form  of  patterns  of  various  types  ... 
Systems  engineers  use  the  general  information  provided  by  patterns  to 
understand  a  specific  system  problem  and  to  develop  a  specific  system 
solution.  (INCOSE,  2015) 

A  variety  of  quantitative  and  qualitative  methods  exist  to  (1 )  capture  or  generate  data 
needed  for  a  particular  analysis,  (2)  reduce  the  data,  (3)  evaluate  the  data  to  find  patterns, 
and  (4)  draw  conclusions  about  the  System  Of  Interest  (SOI).  This  paper  focuses  on  content 
analysis,  a  qualitative  method  that  is  well  suited  for  datasets  that  contain  primarily  text- 
based  data. 

What  Is  Content  Analysis? 

Patton  (2015)  describes  content  analysis  as  “any  qualitative  data  reduction  and 
sense-making  efforts  that  takes  a  volume  of  qualitative  material  and  attempts  to  identify  core 
consistencies  and  meanings.  ...  The  core  meanings  found  through  content  analysis  are 
patterns  and  themes.”  As  defined  by  Krippendorff  (2004),  content  analysis  is  used  to  make 
“replicable  and  valid  inferences  from  texts  (or  other  meaningful  matter)  to  the  contexts  of 
their  use”  and  is  most  successful  when  evaluating  attributions,  social  relationships,  public 
behaviors  and  institutional  realities.  The  basic  steps  to  conducting  a  content  analysis  are 
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summarized  below.  See  either  Krippendorff  (2004)  or  Patton  (2015)  for  detailed  descriptions 
of  each  of  these  steps. 

1 .  Decide  what  data  sources  to  use  for  the  analysis.  These  best  fit  the  research 
questions  (unitizing). 

2.  Identify  a  representative  data  subset  to  analyze  (sampling). 

3.  Transform  the  raw  data  into  analyzable  data;  evaluate  and  interpret 
characteristics  within  and  between  data  elements  by  assigning  elements  to 
categories  based  on  an  observed  pattern  or  theme.  This  can  include  inter¬ 
rater  agreement  studies  for  the  categories  (recording/coding). 

4.  Evaluate  and  interpret  the  categorized  data,  looking  for  additional  patterns, 
themes,  correlations  (e.g.,  sub-categories)  and  outliers  (reducing  data). 

5.  Infer  the  meaning  of  the  categories.  Test  and  validate  the  inferences  with 
respect  to  the  research  questions  (inferring). 

6.  Summarize  and  communicate  the  analysis  findings  (narrating). 

Content  Analysis  in  Systems  Engineering  Activities 

Within  a  systems  engineering  context,  both  “attributes”  of  system  components  and 
“institutional  realities”  with  respect  to  operational  and  maintenance  concepts  for  a  given  SOI 
are  identified  and  evaluated  during  a  system’s  design  lifecycle.  Therefore,  someone  taking 
the  time  to  gather  existing  text-based  documents  from  either  electronic  or  paper  sources 
and  look  for  patterns  and  themes  is  already  done  within  systems  engineering  practice  to 
varying  degrees. 

One  example  is  the  case  where  various  stakeholders  and/or  representative  users  are 
interviewed  to  capture  their  inputs  on  what  the  SOI  needs  to  do  and  what  should  be 
reflected  in  the  corresponding  system  requirements  and  technical  performance  measures. 
The  answers  to  the  interview  questions  have  to  be  evaluated  and  summarized  in  some 
fashion.  Another  example  is  performing  trade  studies,  when  various  industry  information 
sources  are  reviewed  to  understand  the  latest  systems  available  on  the  market  and  current 
technology  trends  that  may  apply  to  the  SOI.  Reviewing  different  documents  or  websites, 
the  systems  engineer  looks  for  very  specific  features  and  compares  and  contrasts  them  in 
some  fashion.  The  INCOSE  (2015)  Systems  Engineering  Handbook  describes,  in  detail, 
each  of  the  standard  technical  processes  and  the  various  activities  that  take  place  within 
each  process.  Table  1  provides  a  sample  of  systems  engineering  activities  described  in  the 
handbook  that  most  likely  involve  the  review  of  qualitative  data  from  text-based  sources. 
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Table  1.  Sample  of  Systems  Engineering  Technical  Process  Activities  for 

Content  Analysis 

(INCOSE,  2015) 


Process 

Activity 

Business  or 

Mission  Analysis 

•  Analyzing  gaps  across  the  problem  or  opportunity  trade  space 

Stakeholder 

Needs  and 

Requirements 

Deinition 

•  Eliciting  and  prioritizing  stakeholder  needs 

•  id  entity  i  n  g  solutio  n  c  o  nstra  i  nts  re  s  u  Iti  n  g  fro  m  a  gree  m  ents  o  r  inte  rf aces  with 
other  systems 

System 

Requirements 

Deinition 

*  Ensuring  the  syste  m  req  u  i  re  m  ents  ad  e  q  u  ately  refle  ct  th  e  sta  ke  ho  Idle  r 
requirements 

•  Negotiating  modifications  to  the  requirements  to  resolve  any  issues  identified 

Architecture 

Deinition 

Process 

•  Analyzing  "relevant  market  industry,  stakeholder,  organizational,  business, 
operations,  mission,  legal,  and  other  information"  to  guide  architecture 
development 

Design  Definition 

*  Identifying  types  of  design  characteristics  for  each  system  element 

Systems  Analysis 

*  Comparing  results  from  several  types  of  models 

Implementation 

•  Eli citi ng  con straints  relat e d  to  i m pie m ent atio n  fro m  "sta ke h old ers ,  de ve  1  ope rs, 
and  teammates" 

*  Analyzing  and  resolving  any  anomalies  that  occurred  during  implementation 

Integration, 
Verification  and 
Validation 

*  Analyzing  and  resolving  any  anomalies  that  occurred  during  integration, 
verification  and  validation 

Operations 

*  Identifying  additional  operations  constraints  and  providing  feedback  to  the 
system  design  team 

•  Analyzing  and  resolving  any  anomalies  that  occur  during  operations 

Maintenance 

*  Identifying  additional  maintenance  constraints  and  providing  feedback  to  the 
system  design  team 

•  Analyzing  and  resolving  any  anomalies  that  occur  during  maintenance  activities 

•  Identifying  trends  in  maintenance  and  logistics  actions 

*  Eliciting  customer  feedback  on  maintenance  and  logistics  support  process 
satisfaction 

Of  particular  interest  to  this  paper  are  the  technical  processes  that  take  place  after 
System  Analysis.  The  Implementation,  Integration,  Verification  and  Validation  processes 
typically  span  the  “Engineering  and  Manufacturing  Development”  acquisition  lifecycle  phase. 
As  described  in  Table  1,  all  of  these  processes  share  one  activity  in  common:  analyzing  and 
resolving  any  anomalies  that  occur  during  each  process’s  execution.  During  this  phase,  the 
product  baseline  for  the  SOI  is  reviewed  and  approved  during  the  key  Systems  Engineering 
Technical  Reviews  (SETRs)  that  are  required  prior  to  the  start  of  the  “Production  and 
Deployment”  acquisition  lifecycle  phase:  the  Critical  Design  Review  (CDR),  Test  Readiness 
Review  (TRR),  and  the  Production  Readiness  Review/Functional  Configuration  Audit 
(PRR/FCA). 

In  preparation  for  each  of  these  SETR  events,  the  contractor  systems  engineers 
evaluate  the  SOI’s  design  and  document  its  status  in  the  required  Contract  Data 
Requirements  List  (CDRL)  documents.  Examples  of  these  documents  are  Design 
Description  documents,  Product  Drawings  and  Associated  Lists  (PDALs),  and  Deficiency  or 
Discrepancy  Reports  (DRs).  These  documents  are  then  reviewed  and  interpreted  by  the 
Government  Systems  Engineers,  Logisticians,  and  Test  &  Evaluation  (T&E)  Engineers  for 
accuracy  and  validity  and  in  order  to  assess  the  SOI’s  adequacy  and  readiness.  Any 
anomalies  would  be  discussed  with  the  contractors  to  either  resolve  or  come  up  with  a 
mitigation  strategy,  preferably  before  the  SETR  event. 
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While  this  may  sound  simple  and  straightforward,  it  can  be  a  daunting  task,  even  for 
systems  with  relatively  few  components  and  for  documents  housed  in  a  configuration 
management  software  like  DOORS®.  Ensuring  consistency  within  and  across  Design 
Description  documents  requires  the  system  engineer  to  cross-reference  the  content  of  each 
document,  looking  for  specific  similarities  and  differences.  This  is  important  because  each 
document  focuses  on  detailed  aspects  of  the  same  SOI.  Similarly,  the  PDALs  may  contain 
hundreds  of  component  and  subsystem  drawings,  including  those  for  the  Commercial  Off 
the  Shelf  (COTS)  components.  The  contractor  systems  engineers  have  to  review  each 
component  drawing,  ensuring  that  the  content  makes  sense  and  correlates  with  the  other 
drawings  that  each  one  references.  The  interfaces  depicted  in  these  drawings  must  also 
match  the  same  interfaces  described  in  the  design  description  documents.  This  matching 
task  can  reveal  configuration  errors  that  could  impact  component  production  in  the  next 
acquisition  phase.  For  the  DRs,  it  is  the  responsibility  of  the  systems  and  T&E  engineers  to 
review  these  reports,  evaluate  them  for  patterns  and  themes,  and  interpret  what  those 
patterns  and  themes  reveal  about  the  performance  of  the  software  and  hardware.  The 
Government  Systems  Engineers,  in  an  acquisition  oversight  role,  independently  repeat  the 
same  process  for  each  one  of  these  documents. 

The  systems  engineer,  as  the  Subject  Matter  Expert  (SME),  will  be  held  accountable 
for  the  hardware  and  software’s  performance  by  the  Program  Manager.  Looking  for  trends, 
correlations,  and  consistencies/inconsistencies  helps  the  systems  engineer  evaluate  the 
feasibility  of  the  technical  baseline  and  qualify  the  reliability  and  quality  of  the  data  used  in 
the  evaluation.  For  the  Government  Engineers,  this  kind  of  analysis  also  helps  gauge  the 
quality  of  the  contractor’s  technical  performance. 

The  Operations  and  Maintenance  (O&M)  processes  described  in  Table  1  correspond 
to  the  “Production  and  Deployment”  and  the  “Operations  and  Support”  acquisition  lifecycle 
phases.  Like  the  previous  technical  processes  discussed  above,  the  O&M  processes  also 
analyze  and  resolve  any  anomalies  that  occur.  Tracking  system  performance  measures  and 
periodically  correlating  that  data  to  deficiency  log  data  or  maintenance  action  reports  can 
reveal  additional  factors  that  are  impacting  system  performance.  Similar  to  the 
Implementation  process,  it  is  important  that  any  additional  constraints  observed  by 
users/operators,  maintainers,  other  engineers  or  stakeholders  within  the  O&M  processes  are 
captured,  documented,  and  evaluated.  Once  fed  back  to  the  system  designers,  this 
information  can  then  be  used  to  shape  requirements  definition  for  system  upgrade  or 
modification  contracts  and  new  baseline  contracts. 

It  is  important  to  note  that  the  systems  engineers  doing  data  analysis  in  the  O&M 
processes  may  be  different  people  than  the  ones  who  worked  on  the  contract  in  previous 
phases  of  the  acquisition  lifecycle.  Instead  of  working  for  the  program  office  or  the  prime 
contractor,  these  systems  engineers  may  work  for  the  installation  site  and  are  responsible 
for  capturing  and  analyzing  system  performance.  Evaluating  and  packaging  these  data  and 
data  analysis  results  can  be  a  different  task  if  it  is  being  done  to  support  local  management 
or  will  be  provided  to  an  outside  organization  for  system  design  purposes. 

Research  on  the  Use  of  Content  Analysis  in  Systems  Engineering  Activities 

Systems  engineers  seem  to  be  performing  some  level  of  content  analysis.  But,  in 
which  technical  activities?  How  “well”  is  it  being  done?  How  valid  are  the  results?  Valuable 
insight  can  be  gained  by  researching  the  actual  use  of  content  analysis  in  the  technical 
processes  previously  discussed  and  what  software  tools  are  used  and  can  be  used  to 
facilitate  the  process. 
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For  example,  Fortune  and  Valerdi  (2013)  developed  a  framework  for  determining 
how  to  reuse  previously  created  engineering  products  for  a  new  development  effort.  As  part 
of  the  evaluation  phase  in  this  framework,  the  first  step  is  to  analyze  both  internally  and 
externally  developed  products  that  are  available,  like  requirements  documents  or  modeling 
tools,  then  determine  whether  or  not  they  apply  to  the  SOI.  Because  this  seems  to  involve  a 
comparison  of  a  previous  system  to  the  new  SOI,  an  investigation  into  the  effectiveness  of 
using  content  analysis  categories  may  prove  to  be  useful.  Since  the  next  step  in  this 
framework  is  to  estimate  the  costs  and  anticipated  benefits  from  reusing  the  engineering 
products,  having  supporting  evidence  generated  from  a  thorough  content  analysis  may  help 
to  justify  the  investment. 

It  is  easily  hypothesized  that  the  successful  use  of  content  analysis  as  a  research 
methodology  within  a  design  environment  or  an  operational  setting  would  be  impacted  by 
factors  such  as 

•  Time  required  and  resource  availability  to  spend  on  the  analysis 

•  Familiarity/Expertise  with  content  analysis  methods 

•  Familiarity/Expertise  with  the  technical  subject  matter  and  data  content 

•  Data  access,  particularly  when  data  are  spread  across  multiple  print  and 
electronic  sources 

•  Data  quality/quantity 

•  Individual  personality — having  the  ability  and  patience  to  search  for  and 
identify  patterns  in  datasets  of  various  sizes 

It  would  be  worth  researching  the  impact  that  content  analysis  would  have  on  the 
system  engineer’s  workload.  Such  a  study  could  provide  supporting  evidence  for  hiring  a 
dedicated  data  analyst  on  acquisition  projects  to  perform  various  technical  content  analyses. 
Investigating  the  degree  to  which  the  other  factors  listed  above  actually  impact  content 
analyses  can  help  identify  constraints  and  possible  mitigations  to  support  the  use  of  this 
methodology  in  different  acquisition  phases. 

Another  possible  research  path  is  the  development  of  valid,  relevant  and  repeatable 
analysis  criteria  that  can  be  used  across  different  system  baselines.  Granted,  every  system 
is  unique.  However,  research  to  develop  either  (1)  appropriate  contexts  and  levels  of  depth 
for  content  analysis  efforts  within  different  acquisition  phases  or  (2)  generalizable 
categorizes  for  system  attributes  would  help  lay  a  foundation  for  integrating  this 
methodology  into  the  systems  engineering  toolkit.  Having  a  common  analysis  tool  that  is 
easy  to  use  would  support  the  feedback  of  observed  system  performance  trends  from  the 
operational  and  maintenance  community  to  the  design  community,  which  would  be  used  to 
develop  requirements  for  system  upgrade  or  modification  contracts  and  new  baseline 
contracts. 

Finally,  the  implications  of  content  analysis  on  training  and  skill  development  for 
systems  engineers  in  the  acquisition  workforce  should  be  investigated.  Frank  (2006) 
evaluated  interview  and  survey  data  (using  content  analysis  as  part  of  his  data  analysis 
methodology)  to  characterize  the  cognitive  characteristics  and  abilities  of  engineers  with  a 
high  Capacity  for  Engineering  Systems  Thinking  (CEST).  While  the  ability  to  identify  patterns 
and  themes  was  not  specifically  identified  in  this  study,  the  characteristic  of  understanding 
analogies  and  parallels  between  systems  and  the  ability  to  conduct  trade  studies  were 
identified.  As  an  analysis  methodology  that  specifically  targets  these  abilities,  it  would  be 
interesting  to  evaluate  use  of  content  analysis  on  the  development  or  enhancement  of  these 
abilities.  It  would  also  be  worth  developing  guidelines  to  use  content  analysis  specifically  in 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-61  - 


baseline  comparison  analyses,  providing  training  on  its  use,  then  determining  any  impact  to 
the  perceived  validity  of  the  analysis  results.  Additional  studies  that  test  instructions  on  how 
to  identify  and  validate  data  sources,  gather  data  from  these  sources,  and  use  commonly 
available  software  tools  like  Microsoft  Excel  would  further  demonstrate  the  feasibility  of 
using  this  methodology. 
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Abstract 

The  Department  of  Defense  (DoD)  acquisition  workforce  is  growing  rapidly,  and  the  need  to 
align  tasks  to  job  positions  and  competencies  with  individuals  to  ensure  positions  are  filled 
with  the  “best  fitting”  candidate  is  extremely  important.  DASN  RDT&E  has  funded  NPS  on  a 
multi-year  project  to  lead  a  multi-agency  working  group  in  the  development  of  a  Systems 
Engineering  Career  Competency  Model  (SECCM).  The  current  phase  of  the  SECCM 
development  project  is  heavily  focused  on  the  verification  of  the  model.  OPM  joined  the 
SECCM  working  group  to  assist  in  the  refinement,  confirmation,  and  strategic  planning 
required  to  ensure  the  systems  engineering  competency  model  is  a  legally  defensible, 
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relevant,  and  sound  tool.  The  analysis  of  the  ongoing  verification  effort  and  an  overview  of 
how  NPS  and  OPM  plan  to  assist  with  DoN  implementation  of  the  SECCM  will  be  discussed. 
Research  results  from  the  SECCM  verification  process  can  be  used  for  key  human  resources 
functions,  such  as  hiring,  promoting,  administering  skill(s)  gap  assessments,  and  in  career 
path  modeling/development  plans. 

Introduction 

The  Department  of  Defense  (DoD)  has  established  a  competency-based  approach  to 
strategic  workforce  management.  This  approach  includes  assessing  the  critical  skills  and 
competencies  needed  now  and  in  the  future  within  the  civilian  workforce,  along  with 
strategies  to  bridge  competency  and  skill  gaps.  A  competency-based  approach  supports 
strategic  workforce  planning  and  effective  talent  management.  The  specifications  of  5 
C.F.R.  300A,  Employment  Practices,  a  federal  regulations  guide,  require  (1 )  a  job  analysis 
for  selection  and  competitive  promotions  in  Federal  employment,  (2)  compliance  with  the 
job-relatedness  requirements  of  the  Uniform  Guidelines  on  Employee  Selection  Procedures 
(43FR38290),  and  (3)  that  resulting  assessments  target  competencies  required  for  the 
occupational  position.  The  Uniform  Guidelines  are  a  set  of  principles  designed  to  assist 
employers,  labor  organizations,  employment  agencies,  and  licensing/certification  boards  in 
complying  with  requirements  prohibiting  discriminatory  employment  practices.  As  such,  the 
Uniform  Guidelines  are  “designed  to  provide  a  framework  for  determining  the  proper  use  of 
tests  and  other  selection  procedures  [in  employment  practices]”  (Biddle  Consulting  Group, 
2015). 

A  DoD  working  group  (WG),  led  by  the  Naval  Postgraduate  School  (NPS),  has  been 
studying  and  refining  the  definition  of  what  competencies  acquisition  workforce  engineers 
must  have  in  terms  of  systems  engineering.  Since  there  is  currently  no  occupational  series 
for  systems  engineering  in  the  U.S.  government,  the  need  to  align  tasks  to  job  positions  and 
competencies  with  individuals  to  ensure  systems  engineering  positions  are  filled  with  the 
“best  fitting”  candidate  is  extremely  important  (Whitcomb,  White,  &  Khan,  2014).  With  these 
thoughts  in  mind,  Deputy  Assistant  Secretary  of  the  Navy  (DASN)  for  Research, 
Development,  Test,  and  Evaluation  (RDT&E)  funded  NPS  on  a  multi-year  project  to  lead  a 
multi-agency  working  group  in  the  development  of  the  Systems  Engineering  Career 
Competency  Model  (SECCM). 

Over  the  past  few  years,  the  SECCM  WG  has  operated  with  members — including  the 
U.S.  Office  of  Personnel  Management  (OPM),  Navy,  Army,  Air  Force,  Marine  Corps,  and  the 
Missile  Defense  Agency — to  develop  and  verify  the  competencies  used  by  defense  systems 
engineers.  OPM  joined  the  SECCM  working  group  to  assist  in  the  refinement,  confirmation, 
verification,  and  strategic  planning  required  to  ensure  the  systems  engineering  competency 
model  is  a  legally  defensible,  relevant,  and  sound  tool.  Within  the  U.S.  government,  only  a 
model  that  is  verified  in  accordance  with  the  Uniform  Guidelines  can  be  used  with 
confidence  for  all  human  resource  (HR)  functions,  especially  for  high  stakes  functions  such 
as  hiring,  selection,  writing  position  descriptions,  and  creating  job  announcements. 
Verification  of  the  competencies  within  the  SECCM  is  critical  to  allow  it  to  be  used  as  a  basis 
for  “high  stakes”  HR  functions  for  all  of  the  U.S.  Department  of  Defense. 

The  SECCM  WG  identified  a  collection  of  knowledge,  skills,  and  abilities  (KSAs)  that 
define  the  basis  for  developing  effective  systems  engineers  that  evolved  over  time  based  on 
availability  of  related  systems  engineering  competency  data.  One  of  the  latest  pieces  of 
information  arrived  in  2016  when  the  Office  of  the  Secretary  of  Defense  (OSD)  released 
their  updated  and  refreshed  competency  model  for  the  engineering  (ENG)  career  field  for 
systems  acquisition.  Their  competency  model  includes  systems  engineer  career 
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professionals  (previously  under  the  Systems  Planning,  Research,  Development,  and 
Engineering  career  field)  along  with  all  other  engineers  under  one  career  field:  ENG.  The 
SECCM  was  subsequently  modified  to  match  the  new  ENG  model.  In  the  current 
configuration  management  controlled  state,  the  SECCM  Baseline  Rev  1  has  3,272  individual 
KSAs  categorized  within  44  competencies.  The  evolution  of  the  SECCM  is  summarized  in 
Figure  1 . 


NPS  +  SECCM  WG 


SECCM  Baseline  Rev  0 


OSD's  ENG 
Career  Field 
r  Competency  Model 
&  Other  Industry/ 
DODSE 
Competency 
Model} 
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C Occupational 
Artifacts 

■  SE  Position  Descriptions 
■  Job  Announcements 
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Facilitation  of  SME  Panels 
Occupational  Analysis 
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Documentation 
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Enylneerine 
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Analysis 
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*  3,272  K$As 

*  44  Competencies 

*  ENG  Model  Update 

*  Tas  k  State  me  nts  from  Su  rvey 


Figure  1 .  Evolution  of  the  SECCM 

The  SECCM  project  focus  is  to  concentrate  on  the  details  for  career  development 
aspects  related  to  the  model  and  the  creation  of  a  “road  map”  to  aid  in  the  implementation. 
The  current  phase  of  the  SECCM  development  project  is  heavily  focused  on  the  OPM 
section  for  verification  of  the  model.  OPM  is  currently  overseeing  the  occupational  analysis 
aspects  as  a  part  of  the  verification  process  of  the  competencies  identified  within  the 
SECCM. 

Many  organizations  within  the  DoD  have  SE  competency  models  that  have  been 
locally  “verified”  or  “validated”  for  their  own  individual  use.  These  uses  include  career 
development,  tracking  education  and  training  requirements,  and  understanding  the  work- 
related  activities  that  systems  engineers  have  to  accomplish.  These  SE  competency  models 
have  been  verified  or  validated  locally  in  the  sense  that  they  have  proven  useful  in  their 
operational  environment  to  define  what  the  respective  systems  engineers  do.  However, 
none  of  these  existing  models  is  currently  verified  IAW  the  Uniform  Guidelines. 

Once  verified,  however,  the  SECCM  can  be  used  to  guide  career  choice  and  self¬ 
selection  by  describing  in  detail  what  is  required  to  be  successful  at  a  particular  job/role.  As 
a  verified  model,  the  SECCM  would  also  assist  human  resource  efforts  to  find  the  “right  fit” 
for  a  position,  as  potential  applicants  would  have  an  informed  understanding  of  what  KSAs 
are  needed  for  a  particular  position  prior  to  applying  for  it.  Furthermore,  as  a  verified 
competency  model,  the  SECCM  can  also  be  used  to  assist  with  leadership  development 
and  career  development  plans.  For  example,  appropriate  training  and  development  plans 
could  be  created  based  on  the  results  of  the  verified  competency  model.  Courses  can  be 
created  to  bridge  specific  competency  gaps  by  developing  specific  competencies. 
Competency  Assessment  tools  could  also  be  derived  to  supplement  academic  qualifications 
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of  applicants  (Patterson  et  al.,  2000).  Competency  models  can  also  be  used  to  evaluate 
employees’  performance,  to  reward  employees  by  using  the  competencies  to  establish 
promotion  criteria  (Morgeson,  Campion,  &  Levashina,  2009),  and  to  manage  employee 
information  by  using  the  competency  models  to  record  and  archive  employee  skills,  training, 
and  job  experience  information.  Employees  could  be  compensated  using  the  model  to 
structure  pay  differences  between  jobs  and/or  to  evaluate  employees  for  pay  increases. 
Retention  of  critical  skills  and  reduction-in-force  activities  can  also  be  managed  through 
identifying  and  measuring  competencies  aligned  to  the  current  and  future  organizational 
objectives  (Campion  et  al.,  201 1 ). 

OPM  Occupational  Analysis  Survey 

OPM  took  a  four-pronged  approach  to  the  job,  or  occupational,  analysis:  review  of 
occupational  information,  facilitation  of  SME  panels,  administration  of  surveys,  and 
documentation.  The  occupational  analysis  methodology  focuses  on  identifying  the 
competencies  and  tasks  that  are  critical  for  employees  functioning  as  systems  engineers. 
This  method  of  analysis  establishes  which  competencies  are  suitable  for  assessment  in 
human  resources  activities. 

OPM  began  the  occupational  analysis  with  a  review  of  the  competencies  along  with 
additional  occupational  information  provided  by  NPS  and  other  DoD  components,  including 
MDA.  The  occupational  information  served  to  further  define  the  competencies.  Adding 
descriptions  to  the  competencies  served  to  ensure  each  competency  included  in  the  model 
is  clear  and  unique.  OPM  also  conducted  an  initial  review  of  the  KSAs  to  refine  the  list.  OPM 
personnel  research  psychologists  facilitated  SME  panels,  removing  and  revising  KSAs  that 
were  not  behaviorally  based  or  measurable  to  ensure  the  resulting  task  statements  had  the 
characteristics  necessary  to  support  a  variety  of  HR  activities  based  on  the  SME  input. 

The  occupational  information  helped  OPM  identify  a  set  of  SE  competencies  and 
draft  task  statements  that  subject  matter  experts  (SMEs)  would  evaluate  during  the  review 
panels.  Panels  were  held  first  with  incumbents  who  currently  perform  systems  engineering 
activities  and  then  with  individuals  who  supervise  those  who  perform  systems  engineering 
activities.  NPS  recruited  SMEs  to  participate  in  the  panels,  requiring  them  to  meet 
experience  criteria  to  ensure  each  participant  had  a  minimum  level  of  familiarity  with 
systems  engineering  activities.  SMEs  provided  input  to  further  revise  competency  definitions 
and  task  statements,  to  identify  competencies  and  tasks  critical  to  systems  engineering, 
which  were  not  represented  in  the  existing  models  researched,  and  to  eliminate  tasks  not 
representative  of  the  job.  The  revised  competencies  and  tasks  served  as  the  foundation  for 
an  occupational  analysis  survey. 

In  preparation  for  the  SECCM  survey  deployment,  the  SE  population  was  needed  to 
assist  in  the  identification  of  those  SEs  to  include  in  the  survey  pool.  Identifying  the 
population  of  systems  engineers  was  a  challenge  for  the  DoN  as  well  as  the  other  defense 
organizations,  as  there  is  currently  no  professional  engineering  occupational  code  or 
position  description  for  SEs  within  the  DoD.  The  SE  population  was  identified  based  on  input 
from  all  participating  organizations.  There  was  no  single  best  way  to  identify  a  systems 
engineer,  so  each  component  was  required  to  identify  their  own  population  based  on 
identifying  those  engineers  who  performed  tasks  related  to  SE. 

The  occupational  survey  was  launched  in  September  2015.  It  was  administered  to  a 
personnel  sample  representing  the  great  majority  of  the  SE  population.  Oversampling  was 
done  to  ensure  a  robust  sample  for  the  results  could  be  used  to  represent  the  population  of 
SEs.  Two  separate  questionnaires  were  developed,  one  for  supervisors  and  one  for 
employees.  OPM  invited  employees  who  perform  systems  engineering  activities  and  their 
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supervisors  to  participate  in  the  survey,  only  retaining  data  from  employees  with  established 
minimum  experience  levels  to  ensure  adequate  familiarity  with  systems  engineering  work. 
Additionally,  survey  branching  methodology  was  used,  which  required  participants  to 
respond  to  questions  designed  to  distinguish  participants  who  function  as  a  systems 
engineer  from  those  who  serve  in  other  engineering  disciplines. 

The  survey  was  sent  to  6,01 1  employees  and  1 ,51 9  supervisors  across  the  DoD. 
Survey  participants  were  asked  to  evaluate  each  competency  and  task  on  criteria  such  as 
frequency,  importance,  required  immediately  upon  entry  into  the  position,  and  need  for 
training.  Figures  2  and  3  show  a  21%  response  rate  for  the  employee  survey  and  6%  for  the 
supervisor  survey.  The  survey  response  rates  increased  with  time  due  to  the  concerted 
effort  the  WG  provided  to  ensure  the  survey  respondents  had  support  from  senior 
leadership. 


Employee  Survey  Respouse 
Total  Surveys  Seat  —  6,011 
21%  Respouse  Rate 


Started  Completed 


Status 


■  October  15  th 

■  October  23  rd 

■  October  27lh 

■  N o'', 'ember  2nd 


Figure  2.  Employee  Survey  Response  Rate  Progression 


Supervisor  Survey  Response 
Total  Surveys  Sent  =  1,519 
6%  Response  Rate 


Started  Completed 


Status 


■  October  15th 

■  October  23rd 

■  October  27th 

■  November  2nd 


Figure  3.  Supervisor  Survey  Response  Rate  Progression 
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In  the  occupational  analysis  survey,  incumbents  indicated  how  frequently  they 
perform  the  tasks.  For  the  competencies,  the  incumbents  rated  importance  and  degree  to 
which  training  in  the  competency  would  help  them  perform  their  jobs  more  effectively.  The 
supervisors  rated  the  importance  of  the  tasks,  and  for  each  competency,  they  rated 
importance  and  degree  to  which  the  competency  is  required  at  entry  to  the  job.  Supervisors 
provided  separate  ratings  of  the  tasks  and  competencies  based  on  the  requirements  for 
incumbents  at  each  grade  level  (GS-07  to  GS-15;  OPM,  2016).  The  survey  was  estimated  to 
take  about  2.5  hours.  Initial  feedback  from  supervisors  suggested  that  their  survey  took 
considerably  longer,  which  could  explain  the  lower  response  rate. 

Department  of  the  Navy  Survey  Analysis 

OPM  started  the  statistical  analyses  for  the  Navy  and  Marine  Corps  survey  data  on 
January  2016.  To  identify  the  critical  tasks,  the  research  psychologists  analyzed  task  ratings 
of  importance  and  the  percent  of  respondents  who  indicated  the  task  is  performed  by  SEs. 
Competencies  critical  for  performing  systems  engineering  activities  were  identified  by 
analyzing  competency  ratings  of  importance.  The  resulting  critical  tasks  and  competencies 
create  the  occupational  profile  for  individuals  performing  systems  engineering  work.  In 
conformance  with  legal  and  professional  guidelines,  OPM  documented  the  methodology  and 
results  for  all  phases  of  the  occupational  analysis.  The  documentation  is  a  necessary 
component  for  demonstrating  the  process  is  sufficient  to  serve  as  a  component  of  a  content 
validation  approach  for  ensuring  the  validity  of  future  human  resources  activities. 

OPM  used  the  results  of  the  survey  to  identify  the  critical  tasks  and  competencies  for 
successful  performance  as  a  systems  engineer  at  the  GS-07  to  GS-15  grade  levels.  The 
survey  was  administered  to  3995  incumbents  and  645  supervisors  from  the  Navy  and 
Marine  Corps.  The  analysis  resulted  in  identifying  a  number  of  critical  tasks  and 
competencies  for  systems  engineers  from  GS-07  to  GS-15,  as  shown  in  Table  1. 

Table  1.  Summary  of  Critical  Tasks  and  Competencies  by  Grade  Level 

(OPM,  2016) 


Grade  Level 

Critical  Tasks 

Critical 

Competencies 

GS-07 

18 

2 

GS-G9 

19 

7 

GS-1 1 

76 

6 

GS-1 2 

165 

16 

GS-13 

174 

36 

GS-1 4 

175 

40 

GS-15 

176 

43 

Of  note  in  Table  1  is  the  lower  number  of  tasks  and  competencies  determined  to  be 
critical  for  grade  levels  GS-07  to  GS-1 1 .  Based  on  conversations  occurring  throughout  the 
SME  panels,  it  is  possible  many  engineers  do  not  enter  into  the  systems  engineering 
profession  until  later  in  their  career  because  they  begin  in  a  specific  engineering  discipline 
and  then  transfer  to  systems  engineering  at  higher  grade  levels.  Therefore,  the  competency 
model  developed  for  systems  engineers  focused  more  heavily  on  technical  competencies 
specific  to  the  systems  engineering  profession  (OPM,  2016). 

In  addition,  OPM  psychologists  analyzed  the  competency  proficiency  data  to  identify 
competency  gaps,  computed  as  the  percentage  of  incumbents  who  rated  themselves  below 
the  required  proficiency  level  identified  by  supervisors.  The  skills  assessment  analysis 
revealed  widespread  skill  gaps  across  the  systems  engineer  workforce.  Navy  can  use  the 
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competencies  identified  in  the  current  occupational  analysis  as  the  basis  for  future  initiatives 
in  any  of  these  areas.  In  addition,  Navy  can  use  the  skill  gaps  identified  across  the  systems 
engineer  workforce  to  identify  and  target  training  and  development  for  systems  engineers 
(OPM,  2016). 

Considerations  for  creation,  formatting,  promulgation,  and  use  are  important,  since 
there  are  considerations  for  which  competencies/KSAs/tasks  can  be  used  for  high  stakes 
human  resource  functions,  as  well  as  other  workforce  career  planning,  development,  and 
shaping  purposes.  OPM’s  analyses  of  the  survey  results  identify  critical  competencies  and 
critical  tasks.  The  Navy  plans  to  promulgate  the  verified  SECCM  (Version  1 .0)  to  all  the 
Naval  components,  Systems  Commands,  and  Warfare  Centers  for  their  use  in  writing 
position  descriptions  and  job  announcements,  drafting  assessment  questionnaires  for  hiring 
actions,  and  using  for  gap  analysis,  employee  analysis,  and  other  human  resource 
functions. 

The  next  phase  of  survey  results  analysis  includes  the  USA,  USAF,  and  MDA,  due 
from  OPM  by  the  end  of  FY  2016.  Once  all  the  survey  results  are  known,  the  SECCM  WG 
can  review  the  results  with  OSD  to  inform  and  offer  its  results  for  possible  use  by  the  entire 
defense  community. 

Summary 

The  SECCM  development  was  led  by  NPS  as  funded  by  DASN  RDT&E.  From  its 
inception  to  FY  2016,  the  project  has  shifted  to  concentrate  on  the  details  for  career 
development  aspects  related  to  the  model.  The  current  phase  of  the  project  is  focused 
heavily  on  the  verification  of  the  model,  which  is  significant  because  without  a  verified 
competency  model,  job  announcements,  position  descriptions,  and  so  forth  cannot  currently 
require  SE  competencies,  knowledge,  skills,  and  abilities.  Unless  a  local  occupational,  or 
job,  analysis  has  been  completed,  they  can  only  desire  them. 

The  results  of  the  survey  analysis  is  a  verified  SECCM.  Furthermore,  the  proficiency 
level  criteria  for  each  individual  competency  at  each  proficiency  level  is  documented  in  the 
model.  Organizations  that  employ  systems  engineers  will  be  able  to  use  the  verified  SECCM 
to  support  their  high  stakes  HR  functions  (i.e.,  job  announcements,  position  descriptions, 
etc.).  Additionally  they  will  be  able  to  develop:  workforce  vectors;  component,  command, 
center,  and  program  workforce  risk  analyses;  workforce  mission/business  case  analysis; 
targeted  training  investment;  and  targeted  enrollment  communication  and  skill  gap  analyses. 

The  SECCM  is  also  informing  graduate  academic  programs  to  specify  student 
outcomes  and  learning  objectives  within  systems  engineering  programs  that  will  ensure  the 
students  have  the  entry-level  KSAs  required  to  perform  successfully  in  their  job.  The 
implications  of  this  research  can  also  be  used  to  develop  structured  curriculum  content, 
assessment,  and  continuous  process  improvement  techniques  related  to  the  development  of 
SE  learning,  and  to  develop  more  valid  and  reliable  instruments  for  assessing  what  systems 
engineers  need  to  learn,  need  to  know,  and  need  to  do  (Khan,  2014). 
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Panel  4.  Strengthening  Schedule  Estimates  in 
MDAPs 


Wednesday,  May  4,  2016 

11:15  a.m.  - 
12:45  p.m. 

Chair:  Rear  Admiral  Thomas  J.  Kearney,  USN,  Director,  Acquisition, 
Commonality,  &  Expeditionary  Warfare,  Naval  Sea  Systems  Command 

Acquisition  Cycle  Time:  Defining  the  Problem 

David  Tate,  Institute  for  Defense  Analyses 

Schedule  Analytics 

Jennifer  Manring,  Principal  Economics  and  Business  Analyst,  The  MITRE 
Corp. 

Thomas  Fugate,  Principal  Acquisition  and  Program  Management  Analyst, 
The  MITRE  Corp. 

Toward  Realistic  Acquisition  Schedule  Estimates 

Raymond  Franck,  Professor  Emeritus,  U.S.  Air  Force  Academy 

Gregory  Hildebrandt,  PhD 

Bernard  Udis,  Professor  Emeritus  of  Economics,  University  of  Colorado 
at  Boulder 

Rear  Admiral  Thomas  J.  Kearney,  USN — is  the  Deputy  Commander,  Naval  Sea  Systems 
Command  (SEA  06),  Acquisition/Commonality/Expeditionary  Warfare.  Kearney  grew  up  in  Dover,  NJ, 
and  enlisted  in  the  Navy  in  1978.  He  was  commissioned  via  the  Villanova  University  Navy  ROTC 
program  in  1984  with  a  Bachelor  of  Science  degree  in  mechanical  engineering.  Additionally,  he  holds 
a  master’s  degree  in  political  science  (international  relations)  from  Villanova  University  and  is  certified 
as  a  Level  III  Program  Manager  from  the  Defense  Acquisition  University. 

Prior  to  command,  his  sea  tours  included  assignments  as  a  Division  Officer  and  Navigation 
Department  Head  aboard  USS  New  York  City  (SSN  696),  Engineer  Officer  aboard  USS  Henry  L. 
Stimson  (SSBN  655  Gold),  and  Executive  Officer  aboard  USS  Helena  (SSN  725),  where  he 
conducted  deployments  and  patrols  to  both  the  North  Atlantic  and  Western  Pacific. 

Ashore  he  served  as  an  NROTC  Instructor  at  Villanova  University,  Executive  Officer/Engineer  Officer 
of  the  Moored  Training  Ship  (MTS  635),  Squadron  Engineer  (Submarine  Squadron  7),  and  as  First 
Commanding  Officer  of  Pre-Commissioning  Unit  USS  Virginia  (SSN  774). 

Kearney  commanded  the  USS  Alexandria  (SSN  757)  from  June  2003  to  December  2005.  During  this 
period,  his  ship  was  awarded  the  Battle  E  for  operational  excellence,  was  runner  up  for  the 
prestigious  Battenberg  Cup  Award  for  top  ship  in  the  Atlantic  Fleet,  and  received  the  Navy  Unit 
Commendation  for  operations  conducted  during  the  first  around-the-world  deployment  via  the  Arctic 
by  a  U.S.  submarine. 

Following  command,  Kearney  entered  the  Acquisition  Professional  Community  in  2006  and  served  as 
the  Deputy  Director  of  the  Navy’s  Test  and  Evaluation  Policy  Office  (OPNAV  N912).  He  then  served 
as  the  Foreign  Military  Sales  Program  Manager  in  the  Undersea  Weapons  Program  Office  (PMS  404) 
and  as  Deputy  Program  Manager  in  the  Submarine  Acoustic  Systems  Program  Office  (PMS  401). 
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Kearney  served  as  the  Program  Manager  for  Undersea  Weapons  and  Targets  from  October  2009  to 
October  2012.  During  this  period,  his  program  was  awarded  a  Secretary  of  the  Navy  Excellence  in 
Acquisition  Award,  and  he  was  the  recipient  of  the  201 1  Naval  Submarine  League’s  Vice  Admiral  J. 
Guy  Reynolds  Award  for  Excellence  in  Submarine  Acquisition.  He  served  as  Vice  Commander  for 
Naval  Sea  Systems  Command  from  June  2013  to  April  2014,  establishing  the  Acquisition, 
Commonality,  and  Expeditionary  Warfare  Directorate  (SEA  06)  as  a  new  directorate  within  NAVSEA. 

His  awards  include  the  Legion  of  Merit  (two  awards),  the  Meritorious  Service  Medal  (five  awards),  and 
various  other  personal,  campaign,  and  unit  awards. 
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Abstract 

Acquisition  cycle  time — roughly  speaking,  the  development  lead  time  to  field  a  working 
system  once  we  have  identified  a  need  for  a  new  materiel  solution — is  a  hot  topic  in  defense 
circles.  This  work  takes  a  data-driven  look  at  historical  and  current  acquisition  program  cycle 
times.  We  show  that  the  trend  toward  longer  cycle  times  over  the  past  few  decades  is 
restricted  to  a  handful  of  high-profile  programs,  which  has  profound  implications  for  effective 
policy  response.  We  also  show  evidence  that  cycle  times  are  driven  by  system  complexity, 
and  that  schedule  slip  is  associated  primarily  with  overly  optimistic  schedule  estimates.  We 
conclude  with  a  discussion  of  the  increasing  importance  of  software  for  development  lead 
times,  and  the  general  problem  of  trading  capability  for  timeliness  in  acquisition  programs. 

Introduction 

The  Cycle  Time  Problem — Perception 

Cycle  times  (or  development  lead  times)  for  defense  weapon  systems  are  a  hot  topic 
in  the  defense  world.  In  September  2014,  the  Under  Secretary  for  Acquisition,  Technology, 
and  Logistics,  the  Honorable  Frank  Kendall,  issued  Version  3  of  the  Better  Buying  Power 
(BBP)  initiative.  One  focus  area  of  BBP  3.0  is  to  “reduce  cycle  time  while  ensuring  sound 
investments”  (Kendall,  2014).  The  lead  article  (Schultz,  2014)  in  the  November-December 
2014  issue  of  AT&L  Magazine  was  entitled  “Please  Reduce  Cycle  Time.”  RAND  produced  a 
2014  report  entitled  Prolonged  Cycle  Times  and  Schedule  Growth  in  Defense  Acquisition:  A 
Literature  Review  ( Riposo,  McKernan,  &  Kaihoi,  2014).  The  House  Armed  Services 
Committee  held  hearings  in  October  2015  on  the  topic  “Shortening  the  Defense  Acquisition 
Cycle.” 

Many  commenters  on  defense  acquisition  have  asserted  that  it  now  takes  too  long  to 
develop  and  field  new  systems,  and  that  we  should  do  something  about  it.  They  say  that  the 
pace  of  technological  advancement,  especially  in  electronics  and  information  technology,  is 
now  so  fast  that  our  “advanced”  military  systems  are  nearly  obsolete  by  the  time  they  are 
fielded.  Even  seemingly  mundane  systems,  such  as  troop  transport  vehicles  and  cargo 
aircraft,  seem  to  take  forever  to  field. 

The  various  stakeholders  in  the  defense  community  have  put  forward  numerous 
theories  for  what  is  causing  the  problem,  many  of  which  place  the  blame  squarely  on 
someone  else.  Worse  yet,  the  appropriate  policy  response  to  the  cycle  time  problem 
depends  sharply  on  which  theory  one  subscribes  to.  Some  observers  diagnose  excessive 
oversight  and  prescribe  a  more  laissez-faire  approach  to  acquisition.  Others  diagnose 
unaffordable  ambitions  and  unnecessarily  demanding  requirements,  and  prescribe  appetite 
suppression  and  fiscal  discipline.  Still  others  diagnose  inept  management  and  excessive 
bureaucracy,  and  prescribe  streamlined  processes  and  organizations. 
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The  Cycle  Time  Problem — Reality 

In  this  paper,  we  will  address  several  important  questions  related  to  the  cycle  time 
problem.  First,  we  will  consider  empirically  whether  there  is  a  general  cycle  time  problem 
across  all  programs,  or  a  specific  cycle  time  problem  within  a  few  programs.  Second,  we  will 
look  at  some  candidate  reasons  why  development  takes  as  long  as  it  does.  Finally,  we  will 
consider  the  overarching  question  of  how  to  decide  what  to  try  to  buy,  and  how  to  try  to  buy 
it,  given  an  understanding  of  how  long  it  is  likely  to  take. 

Cycle  Times  for  Typical  Programs  Are  Not  Increasing 

Figure  1  shows  a  scatter  plot  of  cycle  times1  for  Major  Defense  Acquisition  Program 
(MDAP)  subprograms  (including  initial  development)  as  a  function  of  when  each  system 
reached  its  Initial  Operational  Capability  (IOC),  or  equivalent.  Going  back  to  the  late  1980s, 
there  is  no  apparent  upward  trend.  Statistical  analysis  confirms  that  the  trend  is 
indistinguishable  from  zero,  and  that  the  median  cycle  time  has  been  roughly  eight  years 
over  that  entire  span.  This  absence  of  trend  in  the  median  holds  for  all  commodity  types — 
aircraft,  ground  systems,  space  systems,  ships,  etc. 
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Figure  1.  Program/Subprogram  Cycle  Time  by  IOC  Year 

Why,  then,  is  there  a  perception  that  cycle  times  have  been  getting  worse  and 
worse?  Figure  2  shows  the  same  data,  but  with  each  point  now  proportional  to  the  final  (or 


1  Cycle  time  is  defined  here  as  the  time  in  years  from  program  initiation  to  IOC  or  equivalent.  For  most 
programs,  initiation  is  at  or  just  before  what  is  now  called  Milestone  B.  For  some  programs,  initiation 
is  at  Milestone  (MS)  A,  or  even  earlier  if  the  program  began  as  a  Technology  Demonstration  program. 
For  programs  with  no  formal  IOC,  the  equivalent  milestone  might  be  OPEVAL,  First  Unit  Equipped, 
etc.  The  plot  shows  cycle  times  for  -100  subprograms  for  which  both  cost  and  schedule  data  were 
available. 
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most  recent  estimated)  total  procurement  cost  of  the  system  being  developed,  in  inflation- 
adjusted  dollars. 

Shaded  areas  proportional  to  total  procurement  costs 


CNI  CN  CM  CNI 

Year  of  IOC  (or  equivalent) 

Figure  2.  Cycle  Time  by  IOC  Year  Showing  Relative  Program  Size 

Here,  there  is  a  noticeable  upward  trend  for  the  programs  that  are  spending  the  most 
money  on  procurement.  The  cycle  times  for  these  highly  visible  programs  may  be  driving  the 
perception  that  things  are  taking  longer. 

Cycle  Time  Growth  Is  Getting  Worse  for  Some  Commodity  Types 

The  sense  that  things  are  taking  longer  is  driven  not  just  by  actual  cycle  times,  but 
also  by  cycle  time  growth,  or  “schedule  slip.”  Even  where  programs  are  not  taking  any 
longer  than  they  previously  did,  the  Office  of  the  Secretary  of  Defense  (OSD)  and  Congress 
notice  when  programs  take  longer  than  promised. 

Even  here,  there  is  no  significant  overall  statistical  trend  in  program  schedule  growth 
over  the  past  25+  years.  Figure  3  shows  the  relative  schedule  growth  of  each  of  the 
subprograms  portrayed  in  Figure  1  and  Figure  2,  broken  out  by  commodity  type.  Unlike  the 
overall  cycle  time  story,  here  there  are  significant  differences  among  commodity  types. 
Overall  schedule  growth  trends  are  flat  or  downward  (with  occasional  outliers)  among 
ground  systems,  aircraft,  missiles,  and  ships.  The  trend  is  upward  for  space  systems  and 
Command,  Control,  Communications,  &  Intelligence  (C3I)  systems.  In  addition,  instances  of 
individual  programs  with  schedule  growth  well  above  the  overall  trend  for  their  commodity 
type  are  becoming  more  common. 
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Figure  3.  Percent  Schedule  Growth  by  IOC  Year  for  Various  Commodity  Types 

Of  course,  schedule  growth  does  not  necessarily  indicate  poor  program  execution.  It 
may  be  that  the  program’s  funding  was  cut.  It  may  be  that  requirements  were  changed.  Or,  it 
may  be  that  the  original  schedule  estimate  was  unreasonable. 

We  computed  a  measure  of  relative  schedule  optimism  for  MDAPs,  defined  as  the 
difference  between  the  average  cycle  time  for  programs  of  the  same  commodity  type  and 
the  given  program’s  estimated  cycle  time  at  program  initiation,  divided  by  the  commodity 
average.  Thus,  for  example,  a  program  forecast  to  take  six  years  when  the  average  for  its 
commodity  type  is  eight  years  would  have  a  relative  schedule  optimism  of  (8-6 )/8  =  25%. 
Larger  numbers  indicate  more  optimism;  negative  numbers  indicate  pessimism  relative  to 
the  average.  Figure  4  shows  a  graph  of  cycle  time  growth  versus  this  metric.  We  see  that  a 
clear  relationship  exists  between  schedule  optimism  and  schedule  growth  for  both  new  start 
programs  and  modifications  of  existing  systems.  Interestingly,  the  average  percent  schedule 
growth  for  a  given  level  of  optimism  is  greater  than  the  amount  of  optimism.  This  suggests 
either  that  excessive  optimism  is  a  symptom  of  a  deeper  problem,  or  that  there  are 
cascading  effects  from  being  too  optimistic.  We  will  return  to  that  question  when  we  discuss 
software  development  in  the  section  titled  The  Special  Case  of  Software. 
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Figure  4.  Relationship  Between  Schedule  Optimism  and  Schedule  Growth 

Do  Cancelled  Programs  Make  the  Picture  Look  Better  or  Worse? 

A  confounding  factor  in  any  study  of  MDAP  cycle  times  has  been  the  high  rate  of 
program  cancellations  over  the  past  15+  years.  As  famously  reported  in  the  Decker-Wagner 
report  (201 1  ),2  between  1995  and  2009,  the  Army  spent  roughly  one  quarter  of  its 
Development,  Test,  and  Evaluation  (DT&E)  funding  on  programs  that  delivered  fielded 
capability.  The  other  Services  were  by  no  means  immune  to  this  problem. 

Cancelled  programs  bias  the  statistics  on  cycle  time  by  censoring  data  from 
programs  that  would  tend  to  take  longer  than  average  if  carried  to  completion.  The  effect  of 
individual  cancellations  on  average  or  median  cycle  time  depends  on  why  the  program  was 
cancelled.  Programs  that  were  cancelled  because  they  were  going  to  be  obsolete  by  the 
time  they  were  fielded3  should  have  their  expected  durations  included  in  the  distribution  of 
cycle  time  outcomes  if  our  goal  is  to  understand  the  extent  to  which  programs  take  too  long 
to  be  relevant  or  timely.  Programs  that  were  cancelled  for  other  reasons — for  example, 
technical  infeasibility  or  unaffordability — do  not  generally  tell  us  anything  about  achievable 
cycle  times  for  executable  programs. 


2  The  actual  title  of  this  report  is  Army  Strong:  Equipped,  Trained  and  Ready:  Final  Report  of  the  2010 
Army  Acquisition  Review.  It  is  often  referred  to  as  “the  Decker-Wagner  report,”  after  its  two  co- 
chairmen,  Gilbert  F.  Decker  and  Louis  C.  Wagner. 

3  Obsolescence  can  be  due  to  changes  in  the  threat  environment,  geopolitical  events,  new 
technologies,  etc. 
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However,  cancelled  programs  are  certainly  relevant  to  the  larger  question  of  how 
long  it  takes  to  mitigate  a  capability  gap,  once  it  has  been  identified.  Of  the  major  programs 
cancelled  since  2000,  few  if  any  were  cancelled  solely  due  to  premature  obsolescence.  In 
most  cases  where  obsolescence  was  cited  by  OSD  or  the  Service  as  a  causal  factor,  the 
program  was  also  technically  infeasible  or  unaffordable.  It  is  not  unreasonable  to  think  of  the 
current  Amphibious  Combat  Vehicle  (ACV)  program  as  the  direct  continuation  of  the  earlier 
Expeditionary  Fighting  Vehicle  (EFV)  and  Advanced  Amphibious  Assault  Vehicle  (AAAV) 
programs.  By  that  measure,  the  Marine  Corps  has  been  attempting  to  acquire  a  long-range 
high-speed  amphibious  troop  carrier  since  1995,  and  does  not  expect  to  be  able  to  field  one 
until  2025  at  the  earliest.  In  the  same  way,  the  Army’s  ongoing  attempts  to  replace  the  OH- 
58  Kiowa  scout  helicopter  began  in  1985  with  the  Comanche  program.  Each  Service  has 
outstanding  modernization  requirements  left  unfilled  by  failed  programs;  not  all  of  those 
requirements  have  any  current  MDAP  attempting  to  meet  them.  The  delays  in  fielding  these 
capabilities  may  have  operational  impacts,  but  they  do  not  seem  to  result  from  inefficient 
“acquisition”  in  the  usual  sense. 

Execution  or  Expectations? 

The  historical  cycle  time  data  raise  some  important  questions.  First,  do  long  cycle 
times  reflect  an  acquisition  process  problem  (to  be  addressed  by  changes  to  the  acquisition 
system)  or  an  outlier  problem  (to  be  addressed  by  root  cause  analysis  and  improved 
oversight)?  It  would  probably  be  neither  efficient  nor  effective  to  overhaul  the  entire 
acquisition  system  if  most  programs  are  executing  reasonably  under  the  current  system. 

Second,  is  this  a  problem  of  execution  or  of  expectation?  How  long  should  it  take  to 
develop  a  fifth-generation  fighter  aircraft,  or  a  first-ever  tilt-rotor  transport  aircraft,  or  a  high¬ 
speed  amphibious  assault  vehicle?  Were  the  original  schedule  estimates  implausible? 
Where  do  the  development  schedules  found  in  Acquisition  Program  Baselines  come  from, 
anyway? 

Finally,  to  what  extent  are  our  acquisition  processes  the  pacing  factor  in  MDAP 
development?  Are  there  unnecessary  regulations  that  slow  down  development  without 
adding  value?  Are  there  unnecessary  administrative  processes  (either  within  OSD  or  within 
the  Services)  that  delay  development?  Is  testing  a  cause  of  delay,  or  merely  the  bearer  of 
bad  news?  Are  there  technical  reasons  why  we  should  not  expect  to  be  able  to  go  much 
faster  than  we  currently  do? 

Which  Came  First — The  Program  or  the  Schedule? 

When  deciding  which  new  programs  to  start  and  what  kind  of  system  they  should 
aim  to  produce,  decision-makers  are  informed  by  cost  and  schedule  estimates.  The  National 
Air  and  Space  Administration  (NASA)  goes  so  far  as  to  treat  cost  and  schedule  jointly, 
recognizing  both  that  they  are  highly  correlated  and  that  changes  driven  by  one  will  affect 
the  other.  No  one  is  surprised  when  the  cost  estimates  are  based  on  the  content  of  the 
program — the  capabilities  the  system  is  supposed  to  provide,  the  materials  it  will  use,  the 
maturity  (or  immaturity)  of  the  technologies  to  be  employed,  etc.  OSD  performs  an 
independent  cost  estimate  for  all  major  acquisitions,  which  is  taken  seriously  during 
milestone  deliberations  and  tends  to  offset  some  of  the  more  optimistic  tendencies  of  the 
sponsoring  Services. 

There  is  no  corresponding  attention  paid  to  OSD  concerns  about  schedules, 
however.  Not  infrequently,  the  initial  schedule  estimate  for  an  MDAP  is  not  an  estimate  at 
all,  but  a  constraint  set  externally  with  little  regard  to  program  content  or  historical 
precedent.  Sometimes  this  is  driven  by  anticipated  external  demands  for  a  system  that  is  to 
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be  used  on  multiple  platforms,  as  was  the  case  for  several  of  the  Joint  Tactical  Radio 
System  (JTRS)  subprograms.  Sometimes  it  is  driven  by  a  planned  retirement  agenda  for 
existing  systems,  such  as  the  plan  for  the  Global  Hawk  Block  30  aircraft  to  replace  the  U-2. 
Sometimes  it  seems  to  be  driven  by  impatience;  the  Army’s  never-quite-started  Ground 
Combat  Vehicle  program  was  told  the  delivery  date  of  the  first  production  vehicle  in  its  Initial 
Capabilities  Document  before  even  a  design  concept  had  been  identified. 

How  Long  Should  It  Take? 

No  one  would  argue  with  the  claim  that  a  cost  estimate  should  be  based  on  the 
content  of  the  program,  or  that  it  should  be  informed  by  the  history  of  past  efforts  to  develop 
similar  things.  Surprisingly,  the  analogous  argument  for  schedules  gets  much  less  traction. 
The  link  between  program  content  and  development  cycle  time  has  not  been  as  clearly 
established  among  decision-makers.  There  is  a  lingering  suspicion  that  better  management, 
less  red  tape,  or  less  oversight  would  allow  successful  completion  of  development  projects 
much  more  quickly  than  they  have  been  done  in  the  past. 

There  have  been  some  substantive  studies  that  looked  at  this  question  in  specific 
domains.  Among  the  most  comprehensive  is  Gene  Bearden’s  work  at  the  Aerospace 
Corporation  on  the  cost  and  schedule  drivers  of  space  probes.  Dr.  Bearden  developed  a 
sophisticated  complexity  metric  for  space  probe  projects,  based  on  the  technical  details  of 
the  operational  domain  (earth  orbit  or  planetary),  the  propulsion  technology,  the  required 
data  links,  the  payload  instruments,  etc.  He  then  showed  that  there  is  a  powerful  and 
consistent  relationship  between  project  complexity  and  development  cycle  time.  More 
importantly,  nearly  all  partial  or  complete  mission  failures  occurred  when  NASA  attempted  to 
develop  and  launch  probes  more  quickly  than  the  estimated  required  development  time. 
Figure  5  shows  this  relationship. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-79- 


c 

o 


c 


a. 

o 

s 

9 

Q 


144 

132 

120 

108 

96 

84 

72 

60 

48 

36 

24 

12 

0 


Schedule  as  Function  of  Complexity 

A  Successful 

■  Fa  iled 

■  Impaired 
To-be -determined 


i 


y  =  2  0.0  84  e1-7203* 
R2  =  0.7165 


3* 


a  a  a  A  a  ’  A 

A 

v  *  i  *  - 


4-:X 


/ 


■ 


X, 


x 


0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1 

Complexity  Index 


Figure  5.  Relationship  Between  Space  Probe  Complexity,  Development  Time,  and 

Mission  Success 

(Courtesy  of  and  reprinted  by  permission  of  the  Aerospace  Corporation) 

We  also  investigated  whether  MDAP  cycle  times  are  related  to  either  the  number  of 
Critical  Technology  Elements  (CTEs)  identified  in  the  program’s  Technology  Readiness 
Assessment  (TRA),  or  the  number  of  distinct  program  requirement  records  in  the 
“Performance”  fields  of  the  Defense  Acquisition  Management  Information  Retrieval  (DAMIR) 
system.  Both  are  positively  correlated  with  cycle  time,  although  we  did  not  have  CTE  data 
for  enough  programs  to  establish  statistical  significance.  For  requirement  counts,  there  is  a 
strongly  significant  relationship  with  cycle  time  during  times  of  growing  defense  budgets, 
and  no  relationship  at  all  in  times  of  decreasing  budgets.4  These  results  taken  together 
suggest  that  schedule  is  driven  by  program  content,  but  that  programs  perhaps  shed 
requirements  in  times  of  tight  budgets. 

The  Special  Case  of  Software 

The  connection  between  complexity  and  cycle  time  is  also  well  understood  in  the 
software  industry.  In  his  pioneering  book  The  Mythical  Man-Month,  Fred  Brooks  (1975) 
presented  a  few  key  facts  about  software  development  projects: 


4  Given  that  the  raw  number  of  requirement  records  in  DAMIR  is  only  weakly  correlated  with  actual 
program  complexity,  the  existence  of  a  strongly  significant  (p  =  0.0002)  relationship  between 
requirements  records  and  cycle  time  is  somewhat  surprising.  The  fact  that  this  relationship  is  only 
apparent  during  times  of  growing  budget  is  even  more  surprising.  It  is  possible  that  this  metric  is 
measuring  bureaucratic  complexity  more  than  technical  complexity. 
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•  Adding  staff  to  a  software  project  that  is  behind  schedule  makes  it  take 
longer. 

•  There  is  a  lower  bound  on  the  number  of  defects  in  a  software  system.  The 
more  complex  the  system,  the  higher  this  bound. 

•  Software  development  consists  of  completing  known  work  and  discovering 
new  work.  There  are  fundamental  limits  on  the  efficiency  of  both  aspects. 

•  The  duration  of  a  software  development  project  depends  strongly  on  the 
degree  of  coordination  required  among  the  various  software  modules. 

Lawrence  Putnam  (1978)  used  analogous  reasoning  about  completion  and  discovery 
to  derive  a  quantitative  model  of  project  duration.  He  found  that  total  development  cost  is  a 
function  of  schedule — but  not  in  the  expected  sense  that  a  longer  project  costs  more. 

Rather,  there  is  a  natural  “most  efficient”  schedule  for  a  given  set  of  requirements,  and  any 
attempt  to  accelerate  the  development  to  finish  more  quickly  than  that  natural  duration 
results  in  increased  cost.  Worse  still,  complex  projects  are  only  slightly  compressible;  there 
is  a  sharp  asymptotic  limit  to  how  fast  you  can  try  to  complete  the  project  without  breaking  it. 
Figure  6  shows  this  relationship  schematically.  For  schedules  longer  than  the  natural 
schedule,  cost  increases  roughly  linearly  with  duration  due  to  low  staff  utilization,  as  shown 
by  the  dashed  line.  For  schedules  shorter  than  the  natural  schedule,  staff  utilization  is  high, 
but  completion  outpaces  discovery,  leading  to  inefficient  rework  and  low  quality.  Schedules 
significantly  shorter  than  the  natural  schedule  are  simply  not  possible,  and  development 
efforts  attempting  to  go  faster  than  that  generally  fail. 


Figure  6.  Cost  as  a  Function  of  Duration  for  a  Given  Set  of  Software 

Requirements 

Are  Weapons  Systems  Like  Software  Systems? 

We  have  shown  that  feasible  schedules  for  space  systems  and  for  software  seem  to 
be  constrained  by  program  complexity.  Is  this  true  for  all  weapon  systems?  If  not,  which 
systems  might  it  be  true  for? 

Clearly,  complexity  is  not  a  factor  when  buying  truly  commercial  items.  By  extension, 
complexity  should  not  be  a  factor  when  buying  systems  that  are  minor  modifications  of 
commercial  items,  or  that  use  only  established  commercial  technologies.  This  is  at  the  heart 
of  the  Government  Accountability  Office’s  (GAO’s)  recurring  admonition  that  acquisition  best 
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practice  requires  that  all  critical  technologies  for  a  program  be  mature5  before  MS  B.  It  is 
also  central  to  the  Decker-Wagner  report’s  taxonomy  of  acquisition  risk  categories,  with 
corresponding  development  timelines.  In  that  report,  a  low-risk  acquisition  is  defined  to  be 
one  that  purchases  an  existing  commercial  item  or  modifies  an  existing  system.  A  moderate- 
risk  acquisition  is  defined  to  be  acquisition  of  a  system  that  uses  only  mature,  proven 
technologies.  Even  then,  the  report  cautions  that  you  should  expect  6  to  1 1  years  from 
Materiel  Development  Decision  (MDD)  to  MS  C  if  you  are  developing  a  new  system  that 
does  not  use  exclusively  pre-existing  components  (p.  99). 

However,  maintaining  our  technological  military  advantage  cannot  be  accomplished 
by  only  buying  things  that  already  exist.  The  Decker-Wagner  report  advises  the  Army  to 
manage  risk  in  its  acquisition  portfolio  by  limiting  the  proportion  of  higher-risk  programs  to 
“only  those  [systems]  that  are  truly  urgently  needed  because  they  represent  ‘game¬ 
changing,’  revolutionary  military  capability,  e.g.,  atomic  bomb,  night  vision,  fire-and-forget 
missiles,  and  stealth”  (p.106).  It  also  cautions  that  you  should  expect  an  8-14-year 
development  cycle  (MDD  to  MS  C)  for  such  systems,  even  if  you  do  everything  right. 

Are  Weapon  Systems  Actually  Software  Systems? 

There  is  another,  more  compelling  reason  to  think  that  weapon  system  acquisition 
programs  might  behave  like  software  development  programs — namely,  that  they  are 
software  development  programs.  Nearly  every  MDAP  today  involves  more  software  than 
even  the  most  software-intensive  programs  of  20  years  ago.  The  F-35  aircraft  system,  in  the 
culmination  of  a  trend  begun  more  than  50  years  ago,  has  almost  no  functions  that  are  not 
implemented,  mediated,  or  controlled  in  software. 

We  have  noted  that  there  is  a  natural  minimum  duration  for  a  software  development 
program.  Even  if  it  were  true  that  hardware  development  and  integration  are  no  harder  today 
than  they  have  been  in  the  past,  we  would  expect  there  to  be  a  size  of  software  effort  at 
which  the  software  development  portions  of  the  program  begin  to  dominate  the  schedule. 
Historically,  the  critical  path  for  system  development  has  run  mostly  through  the  hardware 
side  of  the  project.  Software  contributed  vital  capabilities,  but  only  a  small  portion  of  the 
program  cost  or  duration.  As  we  design  systems  that  perform  more  and  more  of  their 
functions  using  software,  that  might  no  longer  be  true. 

In  fact,  there  is  some  evidence  that  we  may  already  be  reaching  the  turnover  point 
where  software  development  drives  schedule  duration.  To  test  the  plausibility  of  this  idea,  I 
used  the  COCOMO-II  software  effort  estimation  tool  to  estimate  the  duration  of  large, 
difficult  (but  not  too  difficult)  software  development  projects,  as  a  function  of  Source  Lines  of 
Code  (SLOC).  I  then  collected  data  on  the  cycle  times  of  recent  software-intensive  MDAPs, 
along  with  their  SLOC  at  IOC.  The  results  are  shown  in  Figure  7. 


5  The  GAO  defines  maturity,  for  this  purpose,  as  a  Technology  Readiness  Level  (TRL)  of  6  or  higher. 
TRL  6  is  defined  as  successful  demonstration  of  a  representative  prototype  operating  in  an 
environment  that  is  operationally  relevant,  given  the  system  requirements. 
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Million*  of  Source  Lines  of  Code 


Figure  7.  Development  Times  of  Recent  MDAPs  vs.  Software  Size  at  IOC 

The  COCOMO-II  prediction  curve  shown  here  is  notional.  COCOMO-II  takes  as  input 
up  to  24  separate  “effort  multiplier”  parameters  and  five  “scale  factors”  related  to  economies 
(or  diseconomies)  of  scale  for  software  projects.  The  curve  shown  here  corresponds  to  a 
software  project  that  is  modestly  above  average  in  all  dimensions.6  It  is  intended  to  provide 
an  empirically  based  estimate  of  the  proper  scaling  of  duration  as  a  function  of  size  for 
software  similar  to  the  kind  found  in  Department  of  Defense  (DoD)  weapon  systems. 
Astonishingly,  without  any  adjustment,  these  initial  parameter  choices  produce  a  curve  that 
seems  to  behave  like  a  tight  lower  bound  on  cycle  time  for  the  systems  in  question.  This 
suggests  that  even  if  software  is  not  already  defining  the  lower  bound  for  MDAP  cycle  times, 
it  soon  will. 

Again,  the  space  systems  community  may  be  slightly  ahead  of  the  DoD  in  reaching 
this  conclusion.  In  a  2009  presentation,  Dr.  Steve  Jolly  (2009b)  of  Lockheed  Martin  Space 
Systems  concluded, 

Software/firmware  can  no  longer  be  treated  as  a  subsystem.  Systems 
engineering  organizations  need  to  engineer  the  software/avionics  system — a 
change  in  leadership  technical  background.  ...  The  game  has  changed  in 
developing  space  systems.  Software  and  avionics  have  become  the  system 
[emphasis  in  original].  Structures,  mechanisms,  propulsion,  etc.  are  all 
supporting  this  new  system. 


6  For  those  familiar  with  COCOMO-II  modeling,  the  curve  shown  in  Figure  7  corresponds  to  an  Effort 
Multiplier  (EM)  of  1 .5  and  an  exponent  (E)  of  1 .1 .  The  EM  reflects  a  development  only  slightly  higher 
than  nominal  in  difficulty  or  complexity  in  all  dimensions.  The  exponent  corresponds  to  a  total  Scale 
Factor  (SF)  of  18,  which  is  very  close  to  the  SF  value  for  a  “nominal”  development. 
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In  a  related  article  for  NASA’s  ASK  Magazine,  Jolly  (2009a)  speculates  that  this  is 
because  software  now  touches  every  part  of  the  system.  It  is  no  longer  an  isolatable 
subsystem  or  module  that  can  be  designed  in  parallel  with  the  system;  it  is  the  common 
element  that  ties  together  all  parts  of  the  system.  The  software  is  the  integrating  element, 
and  as  such  must  be  the  central  feature  of  the  design. 

Not  all  weapon  systems  are  as  software-intensive  as  a  NASA  space  probe — but 
many  of  them  are.  In  particular,  when  the  role  of  software  in  the  program  is  no  longer  such 
that  the  software  can  be  treated  as  a  separable  module,  but  rather  as  an  integrative 
framework,  it  will  be  necessary  to  manage  the  program  as  a  software  development  project 
with  associated  hardware  and  cyber/physical  integration,  rather  than  as  a  hardware 
development  project  with  associated  software. 

Acquisition  Implications 
What  Can  Be  Had  Quickly? 

Putting  together  the  data  regarding  system  complexity,  software  content,  and 
technology  maturity,  we  can  see  that  acquisition  cycle  times  are  bounded  below  by  the 
maximum  of  several  possible  limiting  factors.  If  the  system  is  technically  immature,  we  will 
not  be  able  to  field  it  very  quickly.  If  the  system  involves  very  large  amounts  of  new  software, 
or  highly  integrated  software  and  hardware,  we  will  not  be  able  to  field  it  very  quickly.  This 
would  be  true  even  with  ideal  program  management,  ample  and  stable  funding,  and 
Acquisition  Reform  that  eliminates  all  red  tape  and  oversight. 

So  what  can  we  get  quickly? 

1 .  Truly  non-developmental  items — commercial  products  and  systems  that 
already  exist.  (Think  MRAP.7) 

2.  Upgrades  of  existing  systems  that  insert  already-mature  technology  and  do 
not  overstrain  the  size,  weight,  power,  and  cooling  capacity  of  the  current 
platform.  (Think  UH-60M  or  M1A2  Abrams  upgrades.) 

3.  Integration  of  existing  mature  systems  into  new  capabilities.  (Think 
HI  MARS.8) 

4.  New  systems  developed  using  agile  methods,  in  which  users  (user 
representatives)  work  interactively  with  developers  to  identify  and  evolve  a 
set  of  capabilities  that  are  useful  enough  to  be  worth  fielding,  rather  than 
working  toward  pre-set  Threshold  and  Objective  requirements.  (Think 
CREW.9) 

5.  New  systems  with  extremely  limited  requirements,  where  we  are  willing  to  live 
with  capabilities  at  or  below  current  systems  in  most  dimensions,  in  order  to 
get  enhanced  capabilities  in  one  or  two  urgently  needed  dimensions.  (Think 
F-117A.) 


7  Mine-Resistant  Ambush-Protected  family  of  vehicles. 

8  High-Mobility  Artillery  Rocket  System,  derived  by  mounting  a  Multiple  Launch  Rocket  System 
(MLRS)  launcher  on  a  5-ton  Family  of  Medium  Tactical  Vehicles  (FMTV)  truck. 

9  Counter-radio-controlled  Explosive  Device  program;  a  cumulative  series  of  technology  deployments 
including  (for  example)  the  Thor  III  man-portable  jammer. 
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6.  New  systems  whose  critical  technologies  and  basic  operational 
characteristics  have  already  been  demonstrated  at  TRL  5  or  higher  in  push 
research  and  development  or  demonstration  programs.  (Think  Predator.) 

7.  New  modular  subsystems  that  can  be  used  to  replace  older,  less-capable 
subsystems  on  platforms  that  have  both  modular  architectures  and  sufficient 
design  margin  (e.g.,  size,  weight,  power,  cooling,  etc.)  to  be  able  to 
accommodate  and  integrate  the  new  technology.  (Think  DDG-51 
Modernization.) 

Of  these  approaches,  only  the  final  three  or  four  have  the  potential  to  sometimes 
produce  leap-ahead  technology  advantage.  Each  of  those,  in  turn,  has  its  limitations  and 
caveats. 

In  the  case  of  “limited  requirements”  developments,  operational  effectiveness  is 
liable  to  be  short-lived,  requiring  more  deliberate  follow-on  programs  that  incorporate  the 
leap-ahead  technology  in  a  more  well-rounded  package.  In  the  case  of  R&D  push  from 
technology  demonstration  programs,  someone  has  to  have  had  the  foresight  to  fund  many 
research  and  development  programs  (not  associated  with  major  systems  procurement)  in 
the  relevant  technology  areas  over  the  preceding  decade,  so  that  there  are  mature 
technologies  on  the  shelf  to  choose  among.  Even  then,  the  initial  MDAP  incorporating  those 
technologies  is  liable  to  produce  a  partially  successful  and  fragile  initial  solution  that  will 
need  to  be  followed  up  with  more  robust  designs,  as  was  the  case  with  Predator. 

Rapid  incorporation  of  new  modular  subsystems  on  an  existing  platform  depends  on 
the  existence  of  a  robust,  overdesigned,  modular-architecture  platform  that  can  host  the 
upgrades.  Initial  development  and  deployment  of  those  host  platforms  will  typically  not  be 
especially  rapid,  and  their  initial  capabilities  may  not  be  significantly  better  than  legacy 
platforms.  Their  value  is  in  their  ability  to  enable  future  upgrades  (see,  for  example,  Patel  & 
Fischerkeller,  2013). 

Finally,  in  the  case  of  agile  development,  you  would  have  to  get  very  lucky  to 
produce  a  leap-ahead  capability.  Most  iterative  agile  developments  will  produce  useful  and 
timely  (but  not  revolutionary)  capabilities. 

When  Should  an  Analysis  of  Alternatives  Consider  Cycle  Time? 

The  most  important  decision  in  any  acquisition  is  the  choice  of  what  to  buy.  The 
purpose  of  the  Analysis  of  Alternatives  (AoA)  is  to  provide  decision-makers  with  all  of  the 
relevant  information  regarding  the  cost,  schedule,  and  effectiveness  of  each  of  the  available 
alternatives — including  a  characterization  of  the  risks  in  each  of  those  dimensions.  The 
worst  acquisition  failures  are  the  result  of  choosing  an  alternative  that  is  not  physically 
possible,  or  is  unaffordable,  or  has  very  little  chance  of  being  delivered  in  time  to  be  useful. 

Historically,  schedule  has  not  generally  been  considered  as  either  a  consequence  of 
the  choice  of  alternative,  or  as  a  Key  Performance  Parameter  for  the  program.  In  some 
cases,  this  makes  perfect  sense — most  any  alternative  for  a  peacetime  tactical  wheeled 
vehicle  program  will  take  about  the  same  amount  of  time  to  develop,  and  there  is  little  risk  of 
early  obsolescence  regardless  of  which  design  approach  is  selected. 

For  other  types  of  systems,  decision-makers  should  care  intensely  about  the  trade¬ 
off  between  capability  and  cycle  time.  For  example,  missile  countermeasures  for  aircraft  are 
typically  designed  to  counter  specific  threat  systems.  There  is  little  point  to  a 
countermeasure  development  program  that  is  unlikely  to  produce  any  deployable  system 
until  after  the  anticipated  service  life  of  the  systems  it  counters.  In  addition,  current  capability 
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gaps  in  countermeasures  are  exploitable  today  by  potential  foes.  Even  a  partial  solution 
today  (or  at  least  soon)  would  be  more  valuable  than  a  perfect  solution  10  years  from  now. 

It  seems  only  prudent  that  military  planners  should  have  some  idea  of  the  time 
urgency  of  a  given  system’s  development  and  should  take  that  urgency  into  account  when 
choosing  among  proposed  alternative  system  designs.  Of  course,  this  would  also  require 
both  honest  and  credible  schedule  estimates  for  all  of  the  candidate  alternatives. 

What  About  Maintaining  Technological  Superiority? 

Much  of  the  concern  about  acquisition  cycle  time  that  has  been  expressed  by  DoD 
officials  and  Congress  has  to  do  with  maintaining  the  historical  technological  advantage  of 
the  United  States  in  military  systems.  Our  all-volunteer  force  relies  operationally  on 
capability  overmatch  in  lieu  of  sheer  numbers,  even  as  it  relies  morally  on  exceptional  levels 
of  force  protection  and  defensive  capability.  Given  the  pace  of  advance  for  electronics  and 
cybernetic  systems  in  the  private  sector  and  by  foreign  militaries,  staying  ahead  of  the  curve 
would  seem  to  require  introducing  new  technologies  on  impossibly  fast  timelines. 

As  we  have  discussed  above,  there  are  a  limited  number  of  potential  ways  to 
develop  new  capabilities  on  sufficiently  responsive  timelines.  For  all  but  the  smallest 
systems,  these  methods  depend  on  having  taken  early  and  effective  action,  both  within  and 
outside  the  formal  “acquisition”  process,  in  order  to  be  ready  to  acquire  useful  systems 
quickly  when  the  time  comes.  The  two  most  effective  ways  to  get  leap-ahead  capabilities  on 
short  timelines  are  both  cases  of  technology  insertion.  In  the  first,  a  weapon  system 
developed  as  part  of  a  requirements-free  technology  demonstration  project  forms  the  basis 
for  an  acquisition  program  that  finishes  making  the  system  sufficiently  safe,  effective,  and 
suitable  for  operational  use.  In  the  second,  a  novel  technology — itself  possibly  derived  from 
a  Science  and  Technology  program  or  demonstration  project — is  inserted  onto  an  existing 
platform  or  system.  In  both  cases,  it  is  vital  that  preparatory  actions  (and  spending)  have 
been  made  in  the  past — actions  that  permit  the  new  program  to  avoid  design  false  starts, 
inefficient  concurrent  development  of  platform  and  modules,  and  premature  convergence  on 
suboptimal  designs.  It  would  seem,  then,  that  the  key  to  being  able  to  maintain  technological 
superiority  is  to  have  executed  the  right  set  of  deliberate  actions  in  the  past,  on  a  rolling 
horizon. 

Conclusions 

Why  Haven’t  Cycle  Times  Been  Increasing? 

In  hindsight,  it  is  surprising  that  cycle  times  have  not  shown  a  general  increase  over 
time.  We  know  that  the  systems  we  develop  and  field  have  grown  enormously  in  complexity, 
and  that  this  growth  is  reflected  by  increased  development  cost  and  unit  procurement  cost, 
relative  to  the  legacy  systems  that  we  replace  or  upgrade.  This  is  not  simply  inflation  or  price 
hikes;  the  new  systems  are  much  more  technologically  advanced  than  the  old  ones,  even 
relative  to  the  current  state  of  the  art  in  commercial  systems.  How  then  have  cycle  times 
managed  to  remain  stable? 

One  plausible  answer  is  that  our  design  and  manufacturing  capabilities  have  roughly 
kept  pace  with  the  increased  complexity  of  the  systems  we  procure.  This  is  in  part  a 
tautology,  enforced  by  the  fact  that  we  cannot  build  anything  we  do  not  know  how  to  build. 
Those  systems  that  sought  to  surpass  the  current  state  of  the  art  by  too  much  were 
cancelled,  and  so  do  not  contribute  to  the  observed  statistics. 

This  is  not  a  stable  state  of  affairs.  In  particular,  software  productivity  has  historically 
grown  much  less  rapidly  than  hardware  system  capability,  and  much  less  rapidly  than  the 
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amount  of  software  required  for  our  most  complex  defense  systems.  In  addition,  complex 
software  shows  diseconomies  of  scale,  so  that  development  efficiency  decreases  with  the 
size  of  the  software  to  be  developed.  As  we  shift  more  and  more  of  the  functionality  of  our 
defense  systems  into  software,  we  can  expect  to  see  a  corresponding  increase  in 
development  lead  times. 

Finally,  we  note  that  these  findings  may  have  significant  implications  for  Acquisition 
Reform.  If  there  are  fundamental  technical  limits  to  how  quickly  certain  types  of  systems  can 
be  acquired,  no  amount  of  management  savvy  or  relief  from  regulatory  burden  will  allow  us 
to  acquire  those  kinds  of  systems  any  faster  than  that.  Reducing  cycle  time  would  thus  need 
to  involve  a  combination  of  using  the  “ways  to  acquire  things  quickly”  (enumerated  in  the 
What  Can  Be  Had  Quickly  section)  with  processes  and  regulations  designed  to  facilitate 
those  acquisition  paths.  It  would  also  involve  effective  oversight  to  recognize  what  is  feasible 
as  early  in  the  acquisition  cycle  as  possible,  and  to  choose  acquisition  alternatives 
accordingly. 

Summary 

Looking  back  at  the  past  30  years  of  major  defense  acquisition  programs  (MDAPs), 
we  note  the  following  patterns: 

•  The  median  cycle  time  and  the  distribution  of  cycle  times  for  the  majority  of 
MDAPs  have  been  fairly  constant. 

•  The  number  of  extreme  outliers  from  this  distribution  has  been  growing  over 
time.  Outliers  are  more  frequent  in  the  most  expensive  programs  and  in 
software-intensive  programs. 

•  Cycle  time  growth  has  been  increasing,  especially  in  C3I  and  Space 
programs.  Much  of  this  growth  seems  to  be  associated  with  overly  optimistic 
schedule  estimates. 

•  The  amount  of  software  in  the  most  software-intensive  MDAPs  has  increased 
by  at  least  two  orders  of  magnitude  over  the  period  in  question.  There  is 
reason  to  believe  that  software  (including  software-hardware  integration)  is 
becoming,  or  has  become,  a  schedule-limiting  factor  for  these  programs. 

If  program  complexity  in  general,  and  software  content  in  particular,  are  now  limiting 
factors  for  the  development  lead  times  of  new  systems,  this  has  important  implications  for 
how  we  choose  which  new  programs  to  begin.  When  it  is  important  to  get  new  things 
quickly,  we  will  need  to  test  the  system  designs  that  are  proposed  (using  credible  schedule 
estimates  for  those  designs)  against  our  best  estimates  of  the  urgency  and  useful  life  span 
of  the  capability  being  acquired.  In  particular,  we  need  to  be  aware  when  we  are  asking  for 
an  amount  of  software  that  cannot  plausibly  be  developed  in  the  time  available. 

In  some  cases,  we  will  be  able  to  get  useful  systems  quickly  enough  simply  by 
asking  for  less  initial  capability.  For  those  capability  gaps  where  existing  technologies  are 
not  sufficient,  it  would  be  prudent  to  invest  (on  an  ongoing  basis)  sufficient  resources  in 
broad  technology  development  and  maturation  efforts  to  “keep  the  shelves  stocked”  with 
mature  technologies.  In  parallel  with  those  efforts,  we  would  also  need  to  design  and  field 
(also  on  an  ongoing  basis)  flexible  platforms  that  can  quickly  incorporate  whichever  of  those 
as-yet-unidentified  future  technologies  turn  out  to  be  needed.  The  technology  development 
half  of  that  plan  is  cost-prohibitive  if  we  try  to  do  it  primarily  within  MDAPs;  the  future 
insertion  half  will  not  succeed  if  we  allow  requirements  creep  during  initial  development  to 
trade  away  flexibility  for  immediate  utility. 
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Introduction 

Developing  and  effectively  managing  schedules  is  critical  to  the  success  of  a 
program.  Program  managers  are  becoming  increasingly  aware  of  the  need  for  greater 
accuracy  in  schedule  estimation,  assessment,  and  risk  management  to  control  cost  and 
deliver  on  time.  A  Government  Accountability  Office  (GAO)  assessment  of  86  programs  that 
made  up  the  2012  portfolio  of  Major  Defense  Acquisition  Programs  (MDAPs)  found  that  the 
portfolio  experienced  total  acquisition  cost  growth  of  38%.  In  addition,  the  average  schedule 
delay  in  delivering  initial  capability  was  27  months  when  measured  against  first  full 
estimates  (GAO,  2013),  representing  a  69%  increase  over  a  12-year  period.1  Most  Major 
Automated  Information  Systems  (MAIS)  programs  experienced  schedule  delays  ranging 
from  six  months  to  10  years.  Clearly,  schedule  can  pose  a  significant  risk  and  drive  cost 
growth. 

The  purpose  of  this  research  was  to  help  strengthen  the  acquisition  community’s 
ability  to  produce  data-driven  realism  in  program  schedules.  This  research  effort  had  three 
main  focus  areas:  (1)  compile  schedule  data  from  programs  to  identify  key  schedule  drivers 
and  characteristics  and  build  a  data  repository,  (2)  analyze  the  data  from  statistical  and 
qualitative  perspectives,  and  (3)  document  data  collected  and  analysis  performed,  and  how 
it  can  be  accessed  for  analysis. 

The  detailed  approach  used  in  the  research  was  comprised  of  the  following  high- 
level  steps: 


Identify  and  review  primary  data  sources 


1  Calculated  based  on  GAO  data  that  the  average  schedule  delay  in  delivering  initial  capabilities  was 
16  months  in  2000  and,  compared  to  21  months  in  2007,  represents  a  31%  increase  in  schedule 
delays  over  a  seven-year  period.  See  the  2008  GAO  report  on  95  weapons  systems  cited  by  Meier 
(2010). 
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•  Develop  a  list  of  program  attributes  to  evaluate 

•  Develop  an  Excel-based  data  collection  framework 

•  Collect  data  and  populate  data  repository 

•  Synthesize  and  cleanse  data 

•  Analyze  and  assess  data 

•  Develop  findings 

•  Document  research 

The  initial  focus  of  the  data  collection  phase  was  to  identify,  understand,  and  review 
existing  external  data  repositories  or  sources  that  collect  enterprise-level  acquisition 
information  and  data,  particularly  related  to  program  schedule.  Across  the  federal 
government,  the  Department  of  Defense  (DoD)  sources  and  the  Office  of  Management  and 
Budget  (OMB)  Information  Technology  (IT)  Dashboard  were  data  sources  identified  and 
reviewed  for  this  research. 

Based  on  the  data  source  review,  the  study  team  determined  that  the  Defense 
Acquisition  Management  Information  Retrieval  (DAMIR)  was  the  best  source  of  data  for  DoD 
large  scale  programs  as  it  contained  both  MDAP  and  MAIS  program  data  (DoD,  2015). 
Although  the  DAMIR  data  is  high  level,  it  provided  sufficient  schedule  fidelity  and  additional 
cost  and  program  parameters  of  interest  to  be  useful  for  the  research.  However,  detailed 
reviews  of  the  OMB  IT  Dashboard  data  revealed  that  schedule  data  is  highly  aggregated. 
Program  start  date  and  program  end  date  are  the  only  schedule  parameters  included  with 
no  intermediate  schedule  milestone  data  available  in  the  repository.  The  OMB  IT  Dashboard 
has  a  large  amount  of  data;  however,  without  further  fidelity  and  definition  for  schedule  data, 
it  was  determined  that  it  would  not  be  useful  to  this  research.  No  additional  enterprise-wide 
acquisition  data  sources  with  relevant  schedule  data  were  identified;  however,  there  are 
likely  other  centralized  systems  that  contain  acquisition  information  but  are  not  open  source. 

Ground  rules  and  assumptions  (GR&As)  were  developed  to  bound  and  scope  the 
research  and  establish  baseline  conditions  for  the  analysis.  Key  GR&As  included  the 
following: 

1 .  Only  actual  data  was  analyzed.  Future  schedule  dates  were  excluded  from 
the  analysis. 

2.  Milestone  equivalents  were  assumed  to  compare  data  between  older 
program  milestones  and  new  program  milestones,  e.g.,  MS  II  =  MS  B. 

Programs  with  negative  or  zero  schedule  durations  between  milestones  were 
removed  from  the  analysis  as  these  reflect  acquisition  process  anomalies.  Negative  duration 
values  could  occur  either  when  the  program  had  unique  circumstances  that  caused 
milestones  to  take  place  in  non-sequential  order  or  could  represent  an  error  in  the  data. 

The  study  team  developed  an  Excel-based  data  framework  that  included  schedule 
parameters,  cost  parameters,  and  program  attribute  parameters  that  were  both  available 
and  of  interest  to  analyze.  Cost  by  appropriation,  major  schedule  milestone  dates,  and  life 
cycle  phase  were  collected.  Program  attributes  collected  included  program  type,  assigned 
component,  acquisition  category,  Joint  Capability  Area  (JCA),  new  start  or  modification,  and 
whether  a  breach  occurred  within  the  program.  Additionally,  program  baseline  costs  were 
synchronized  into  constant  year  15  millions  of  dollars  (CY15$M)  to  avoid  any  dollar 
distortions  in  the  data.  Schedule  durations  between  milestones  were  calculated  (in  months) 
and  included  in  the  repository.  The  analysis  focused  on  analyzing  predictors  of  schedule 
duration. 
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In  total,  the  study  team  analyzed  more  than  2,600  data  points,  including  560 
schedule  milestone  dates  from  143  MDAP  and  MAIS  programs  from  DAMIR.  Table  1  shows 
a  summary  of  the  data  points  collected.  Within  the  schedule  data  points,  the  largest  data  for 
MDAP  and  MAIS  programs  were  for  MS  B  and  MS  C.  The  data  had  all  Services 
represented,  with  Navy  and  Air  Force  having  slightly  more  data  points.  Lastly,  the  data 
showed  each  of  the  JCA  areas  represented,  with  Force  Application  having  slightly  more 
data  points.  Although  the  data  was  high  level,  it  provided  sufficient  schedule  fidelity  and 
additional  cost  and  program  parameters  of  interest  to  have  value  for  the  research. 

Table  1.  Data  Collection  Summary 


The  data  analysis  began  with  synthesizing  and  cleansing  the  collected  data.  The 
next  phase  was  characterizing  the  data.  The  size  and  makeup  of  the  data  by  the  different 
parameters  collected  was  analyzed  for  insights,  trends,  and  relationships.  The  analytic  tool 
was  Excel  with  data  analysis  add-in  features  as  well  as  graphing  capability.  Scatter  plots, 
trend  lines,  and  various  statistical  analysis  were  conducted  to  gain  insight  into  what 
relationships  the  data  may  reveal.  Additional  ways  of  analyzing  the  data,  such  as  changes 
over  time,  were  also  performed  for  further  insight. 

Among  the  key  findings  was  a  wide  range  of  variability  and  a  lack  of  strong  linear 
relationships  between  schedule  durations  and  program  attributes  analyzed.  This  implies  that 
the  complexity  of  DoD  large-scale  programs  was  not  easily  explained  by  predictive 
parameters  such  as  cost,  program  type,  JCA,  and  Service,  for  example.  Despite  this,  the 
research  revealed  several  emergent  trends.  Presented  in  this  summary  are  two  emergent 
trends: 


Trend  1:  Average  MS  B  to  MS  C  durations,  which  accounts  for  the  Engineering  and 
Manufacturing  Development  (EMD)  Phase  of  the  DoD  acquisition  life  cycle,  have  decreased 
43%  for  MAIS  and  42%  for  MDAPs  over  the  last  25  years,  as  shown  in  Figures  1  and  2.  This 
reduction  in  the  EMD  phase  average  duration  was  not  easily  explained  by  the  data,  but 
suggests  that  various  efforts  to  improve  acquisition  outcomes  may  have  contributed  to 
reduced  development  schedules.  For  example,  in  the  last  25  years,  large  acquisition 
programs  have  trended  away  from  single  pass  (aka,  “Big  Bang”)  efforts  in  favor  of 
incremental  development  and  delivery  of  needed  capabilities. 
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Figure  1.  MDAP  Development  Times 
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Figure  2.  MAIS  Development  Times 

Trend  2:  From  a  cost  correlation  perspective,  the  study  team  observed  that  higher 
Research,  Development,  Test,  and  Evaluation  (RDT&E)  costs  as  a  percentage  of  total 
Acquisition  Costs  correlated  with  shorter  EMD  schedule  durations  for  both  MAIS  and  MDAP 
efforts.  This  observation  suggests  that  development  schedules  may  be  “bought  down”  with  a 
greater  share  of  the  total  acquisition  budget,  as  depicted  in  Figures  3  and  4. 
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Figure  3.  MDAP  RDT&E  Percentage  vs.  Schedule  (MS  B  to  MS  C) 


Figure  4.  MAIS  RDT&E  Percentage  vs.  Schedule  (MS  B  to  MS  C) 

The  data  collected  during  this  research  can  be  used  to  help  validate  schedule 
realism  and  identify  schedule  outliers  compared  to  major  DoD  programs  with  similar 
program  attributes.  For  more  comprehensive  analysis  and  higher  confidence  predictive 
measures,  additional  parameter  analysis  such  as  requirements  and  funding  stability, 
program  office  maturity,  governance  structures,  and  technology  maturity  is  needed.  Also,  as 
noted,  the  dataset  covered  only  large  DoD  programs.  The  study  team  did  not  find  useful 
data  for  civil-sector  and  below  ACAT-I  threshold  DoD  programs. 
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In  summary,  the  data  suggested  some  emergent  trends;  however,  the  data  had  a  lot 
of  variability  and  a  wide  range  for  most  schedule  milestones.  The  data  also  does  not  appear 
to  have  strong  linear  relationships  between  schedule  milestone  length  and  program 
attributes  analyzed  for  either  MAIS  or  MDAPs.  One  explanation  is  that  the  complexity  of 
large-scale  DoD  programs  is  not  easily  explained  in  full  by  the  selected  predictive 
parameters.  Additional  parameter  analysis  is  needed  to  explain  and  predict  schedule 
differences  (such  as  funding  stability,  program  manager  experience,  requirements  change, 
technology  readiness  levels,  etc.),  but  these  parameters  may  be  difficult  to  collect.  However, 
the  data  collected  is  a  rich  dataset  that  can  be  used  for  analogy  comparisons  for  large  DoD 
programs.  The  study  team  recommends  the  data  be  used  to  perform  high-level  schedule 
assessments  or  for  analogies  to  evaluate  schedule  realism  throughout  a  program’s  life 
cycle. 
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Abstract 

Time  needed  to  develop  and  field  new  military  capabilities  is  becoming  an  increasingly 
serious  problem.  Among  other  things,  development  times  have  steadily  increased.  In  this 
paper,  we  attempt  to  structure  the  schedule  estimating  problem,  present  some  initial  results, 
and  propose  a  research  agenda  to  improve  schedule  estimating.  Accordingly,  we  seek 
preliminary  answers  to  the  following  questions. 

•  What  is  the  current  state  of  the  art  for  estimating  acquisition  program  schedules? 
What  should  it  be? 

•  What  are  salient  features  of  program  management  trade-offs,  especially  between 
schedule  and  cost  (which  are  related  in  complex,  imperfectly  understood  ways)? 
In  what  areas  should  air  combat  performance  measures  need  updating? 

•  What  are  the  elements  of  a  research  agenda  for  learning  more  about  schedule 
estimating? 

We  also  present  some  preliminary  results  in  the  form  a  narrative  case  study  of  the  F-35 
program  and  empirical  estimates  of  schedules. 

The  JCIDS  (Joint  Capability  Integration  and  Development  System)  Instruction  recommends 
“effective  cost,  performance,  schedule  and  quantity  trade-offs”  as  being  highly  conducive  to 
successful  acquisition  programs  (CJCS,  2015,  p.  A-9).  However,  these  attributes  have 
received  rather  unequal  interest — with  cost  garnering  the  most  attention. 

For  example,  in  the  DoD’s  latest  acquisition  performance  report  (DoD,  2015),  “cost”  appears 
18  times  in  the  table  of  contents  and  86  times  in  the  highlights;  “schedule”  appears  six  times 
in  table  of  contents  and  37  times  in  the  highlights.  (“Operational  performance”  appears  only 
six  times  in  the  contents.)  In  our  conference  program,  “cost”  appears  14  times  in  five 
sessions,  while  “schedule”  appears  four  times,  in  one  session. 
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Cost  is  certainly  important,  and  warrants  much  attention,  which  it’s  received.  The  DoD  has 
devoted  considerable  time,  attention  and  resources  to  more  realistic  cost  estimating.  And, 
seems  to  us,  there’s  been  a  great  deal  of  progress  toward  that  goal. 

But  schedule  is  also  important  and  is  becoming  more  so.  With  multiple  Revolutions  in  Military 
Affairs  ongoing  simultaneously,  we  have  entered  a  hyper-adaptive  era  in  military  affairs — 
enabled,  inter  alia,  by  rapid  advances  in  information  technology.  As  Deputy  Secretary  Work 
has  stated,  innovators  now  encounter  “fast  followers”  (Freedburg,  2015).  Accordingly,  the 
operational  implications  of  longer  schedules  have  indeed  become  more  important.  And  the 
DoD’s  leadership  recognizes  that  importance.  A  number  of  organizations  have  recently  been 
created  in  order  to  field  new  capabilities  more  quickly. 

In  short,  major  changes  in  international  military  affairs  and  recent  DoD  emphasis  has  created 
a  new  environment,  in  which  an  ability  to  understand  the  schedule  consequences  of  program 
strategies  is  especially  important. 

Accordingly,  we  discuss  the  matter  of  acquisition  schedules  within  the  context  of 
contemporary  military  affairs  below.  Then  we  address  schedule  estimating  tools  and 
schedule  estimating  methods.  In  the  section  following  that,  we  take  the  JCIDS  instruction 
literally,  and  essay  an  abstract  discussion  of  cost-time-performance  trade-offs. 

One  promising  variable  for  schedule  estimating  relationships  is  system  performance,  which  is 
discussed  next — primarily  in  the  context  of  tactical  fighters,  the  F-35  in  particular. 

The  section  titled  Toward  Explaining  the  Time  Curve1  is  about  empirical  models  for 
estimating  schedules.  One  likely  schedule  driver  is  requirements  growth.  In  a  following 
section,  we  offer  a  narrative  concerning  the  requirements  growth  that  occurred  from  CALF 
(Common  Affordable  Lightweight  Fighter)  to  JSF  (Joint  Strike  Fighter,  F-35).  Finally,  we  offer 
concluding  comments  and  thoughts  about  a  research  agenda  aimed  at  making  more  realistic 
schedule  estimating  tools  available  to  our  acquisition  professionals. 

Introduction 

Why  Schedules  Are  Important 

“The  fact  is  that  we  are  slower  than  the  bad  guys.” 

— Esti  Peshin,  Director  of  Cyber  Programs  for  Israel  Aerospace  Industries 
(quoted  in  Sternstein,  2015). 

Cost  and  schedule  are  critical  variables  in  any  acquisition  program.  And  the  DoD  has 
indeed  committed  serious  efforts  over  an  extended  period  of  time  to  develop  the  means  for 
realistic  acquisition  cost  estimates. 

There  are  at  least  five  good  reasons  for  increased  focus  on  realistic  schedule 
estimates: 

•  Planning  to  pay  for  force  modernization  in  an  era  of  restrained  budgets, 
especially  in  the  next  decade; 

•  Longer  times  to  field  new  capabilities  (absolute  and  relative); 

•  Rapid  fielding  initiatives  throughout  the  DoD;  and 


1  That  is,  the  upward  trend  in  time  needed  to  field  new  systems. 
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•  Current  marching  orders,  including  the  Air  Force  “should-schedule”  initiative. 

(Apparently)  Looming  Budget  Squeeze 

There’s  a  budget  squeeze  for  the  DoD  expected  in  the  2020s,  driven  largely  by 
modernization  programs.  This  has  been  well  documented  by  experts  both  inside  and  outside 
the  government  (e.g.,  Congressional  Research  Service  [Gertler,  2015];  Center  for  Strategic 
and  International  Studies  [Harrison,  2016]). 

This  is  especially  difficult  for  the  Air  Force,  whose  top-3  priority  acquisition  programs 
account  for  almost  all  of  resources  expected  to  be  available  for  modernization.  For  example, 
these  (KC-46,  F-35,  Long-Range  Strike  Bomber),  plus  C-130J  and  unmanned  aircraft 
account  for  99%  of  the  service’s  aircraft  acquisition  budget  for  FY16  (Gertler,  2015, 
Summary,  1),  with  the  situation  continuing  throughout  the  2020s. 

Furthermore,  there’s  every  reason  to  expect  budget  squeezes  to  continue  well  into 
the  future.  The  entitlement  bills  are  now  coming  due — expected  to  account  for  about  15%  of 
GDP  in  2026.  Net  interest  on  the  federal  debt  is  estimated  at  3%;  “discretionary” 
expenditures  for  about  5%,  about  half  of  which  is  estimated  for  defense  (Congressional 
Budget  Office  [CBO],  2016,  esp.  pp.  66,  84).  Revenues  are  estimated  at  about  18%  of  GDP 
(CBO,  2016,  p.  92),  with  long-term  deficits  of  about  5%  of  GDP.  This  means  major  pressure 
on  the  “discretionary”  categories — defense  especially. 

Therefore,  some  preplanning  and  painful  prioritizing  seems  both  necessary  and 
inevitable.  A  number  of  options  are  available,  none  of  them  pleasant  (Gertler,  2015;  Hale, 
2016;  Harrison,  2016).  And  it’s  reasonable  to  expect  that  the  longer  necessary  decisions  are 
postponed,  the  range  of  alternatives  available  will  continue  to  narrow.  But  without 
reasonably  good  program  schedule  estimates,  any  early  decision  loses  credibility  and 
usefulness. 

Schedules  Have  Become  More  Important:  Time  to  Deliver  New  Systems  Is  Increasing 

“(Acquisition)  lead  time  in  the  U.S.  is  too  long,”  according  to  LTG  Arthur  Trudeau, 
Army  Chief  of  Research  and  Development  (1958,  quoted  in  Peck  &  Scherer,  1962,  p.  425). 
But  lead  times  are  getting  longer.  For  example,  the  F-35  concept  is  generally  regarded  as 
being  formed  in  July  1993,  with  the  creation  of  the  Joint  Advanced  Strike  Technology  (JAST) 
program  (Defense  Science  Board  [DSB],  1994,  ES-1).  The  F-35B,  for  example,  was 
declared  operational  on  July  31, 2015  (USMC,  2015),  meaning  a  lead  time  of  22  years. 

This  is  illustrated  in  Figure  1 .  One  implication  is  that  the  widely-mentioned  2030  IOC2 
of  a  next-generation  fighter  aircraft  appears  fanciful  at  best.  If  source  selection  for  a  sixth- 
generation  fighter  aircraft  occurs  in  2020  (optimistic),  we  can  expect  an  IOC  some  time  past 
the  middle  of  the  2030s. 

Schedules  Are  Becoming  More  Important  in  an  Era  of  Faster  Followers 

As  Robert  Work,  Deputy  Secretary  of  Defense,  put  it,  we’re  in  an  era  of  “fast 
followers”  in  military  affairs  (Freedburg,  2015).  And  there’s  excellent  reason  this  problem  is 


2  Fielding  a  new  fighter  in  2030  has  been  advocated  as  an  operational  “requirement”  (Gen  Mike 
Hostage,  quoted  in  Mehta,  2012)  and  also  to  alleviate  fighter  aircraft  shortfalls  (Tirpak,  2009,  38). 
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getting  worse;  as  an  Air  Force  flag  officer  put  it,  “Emerging  threats’  timelines  are  decreasing. 
(Our)  acquisition  times  are  increasing.”3 

The  current  schedule  difficulty  is  made  more  acute  due  to  the  adaptiveness  of  our 
rivals.  Given  especially  the  extended  period  of  development  for  many  U.S.  weapon  systems, 
those  countermeasures  have  time  to  development.  Thus,  for  example,  potential  adversaries 
have  seriously  pursued  countermeasures  to  U.S.  stealth  fighters  (e.g.,  Fulghum,  2012; 
Keller,  2016;  Majumdar,  2014;  Sweetman,  2015c,  201 5d).  The  principal  enabling  technology 
is  rapid  computing,  which  can  combine  fragmentary  sensor  information  into  a  unified  picture 
(Clark,  2014).  All  in  all,  stealth  may  indeed  be  overrated  (as  the  CNO,  Admiral  Jonathan 
Greenert,  stated,  quoted  in  Hasik,  2016a). 


SOURCE  SELECTION  YEAR 


Figure  1 .  The  Time  Curve  in  Months  to  IOC  vs.  Source  Selection  Year  for  Tactical 

Fighters 

(expanded  from  Blickstein  et  al.,  201 1 ,  Table  4.5,  p.  48) 

This  is  relevant  to  schedules.  It’s  possible  that  a  stealthy  aircraft,  if  delayed  long 
enough,  can  operate  only  at  considerably  reduced  effectiveness  (Franck  et  al.,  2012,  p.  68). 
Therefore,  it’s  important  that  new  capabilities,  and  upgrades,  be  fielded  in  a  timely  manner 
and  that  planners  have  a  realistic  estimate  of  how  long  it  will  take  to  field  new  combat 
capabilities. 

Current  Marching  Orders  Regarding  Schedules 

Recognizing  the  problems  discussed  above,  the  services  and  the  DoD  have 
undertaken  initiatives  to  field  new  capabilities  sooner.  These  have  included  the  Air  Force 
Office  of  Rapid  Capabilities  (RCO)  and  OSD’s  Strategic  Capabilities  Office  (SCO).  The  RCO 
began  in  2003;  its  basic  purpose  is  to  accomplish  “expedited  and  operationally  focused 
concept-through-fielding  activities  to  support  immediate  and  near-term  needs”  (Clark  & 


3  Observation  offered  at  a  symposium  in  May  2015,  not  for  attribution. 
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Freedburg,  2016;  U.S.  Air  Force,  2009).  In  addition,  the  Navy  has  proposed  an  office  which 
will  “be  something  that  closely  mirrors  the  Air  Force  RCO,”  according  to  the  Navy’s  senior 
acquisition  officer  (Clark,  2016). 

The  SCO  originated  in  2012.  Its  basic  purpose  is  “to  re-imagine  existing  DoD  and 
intelligence  community  and  commercial  systems  by  giving  them  new  roles  and  game¬ 
changing  capabilities  to  confound  potential  enemies  (with)  the  emphasis  ...  on  rapidity  of 
fielding”  (Carter,  2016;  Clark  &  Freedburg,  2016).  In  addition,  an  ongoing  legislative 
initiative,  associated  with  Rep.  Mac  Thornberry  (R-TX),  is  intended  to  streamline  acquisition 
processes  (Hasik,  2016b). 

Also,  Secretary  of  the  Air  Force  Deborah  Lee  James  (2015)  has  begun  the  “should 
schedule”  initiative:  “The  previous  incentive  focused  on  cost,  now  we‘d  like  to  target  delivery 
time.  ...  If  we  can  collectively  beat  the  historical  developmental  schedules  and  reward  the 
behavior  in  government  and  industry  that  speeds  things  up,  we  have  a  real  chance  to  make 
a  difference.” 

To  implement  the  “should  schedule,”  Secretary  James  proposes,  inter  alia,  that 
schedule  be  a  major  factor  in  source  selections:  “If  an  industry  partner  can  propose  a 
solution  that  credibly  offers  a  way  to  accelerate  successful  EMD,  then  that  company  would 
have  a  competitive  advantage  for  the  award”  (James,  2015). 

This  sounds  good,  but  let’s  consider  a  future  acquisition  scenario.  Suppose  a  major 
acquisition  program  involves  proposals,  from  Firms  A  and  B,  and  that  Firm  A  wins  the 
competition.  Let’s  further  suppose  that  estimated  schedule  is  a  major  factor  in  that  decision. 
Finally  suppose  this  particular  program  involves  a  long-term,  high-value,  winner-take-all 
contract — like  many  competitions  these  days.  And  it’s  a  safe  bet  that  Firm  B  will  protest.4 
While  “any  accelerated  EMD  plan  would  need  to  survive  a  detailed  scrub  by  independent 
engineers”  (James,  2015),  that  might  well  not  be  enough.  At  minimum,  those  proposed 
schedules  should  also  survive  a  detailed  scrub  by  the  GAO. 

On  Schedule  Estimating  Methods 

Program  schedule  time  can  be  analyzed  and  forecast  according  to  the  following 
(non-inclusive)  menu: 

•  Schedule  length  arising  from  an  orderly  relationship  involving  key  variables; 

•  Schedule  as  a  result  of  a  series  of  management  decisions  intended  to 
produce  the  best  outcome  with  respect  to  performance,  cost  and  time; 

•  Schedule  resulting  from  the  interactions  among  a  set  of  tasks  needed  to 
complete  the  program. 

We  know  quite  a  bit  about  the  last  item — through,  for  example,  Program  Evaluation 
and  Review  Technique,  Critical  Path  Method  and  Gantt  Charts  (Blanchard  &  Fabrycky, 
2006,  esp.  Chap.  11;  Defense  Systems  Management  College  [DSMC],  2001). 


4  The  protest  may  or  may  not  have  a  convincing  rationale.  See,  for  example,  Bill  Sweetman’s  (201 5e) 
analysis  of  the  Boeing-Lockheed  Martin  protest  of  the  LRSB  source  selection. 
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We  know  less  about  the  second  but  can  learn  more  through  case  studies  (as 
discussed  below),  and  official  post-mortems  like  those  conducted  for  cost  problems  by  the 
OSD’s  Office  of  Program  Assessment  and  Root  Cause  Analysis  (PARCA)  office.5 

We  know  some  things  about  the  first,  through  descriptive  analyses  such  as  illustrated 
in  Figure  1 .  And  we  can  do  more  to  improve  empirical  methods.  Along  those  lines,  the 
discussion  below  provides  an  interesting  empirical  analysis  of  schedule  lengths. 

It’s  possible  to  formulate  an  orderly-relations  approach  in  a  manner  similar  to 
formulating  Cost  Estimating  Relationships.  We  offer  the  term  “Schedule  Estimating 
Relationships.”  We  already  have  a  fair  number  of  possibilities  for  key  explanatory  variables. 
These  include  the  following: 

•  risk  reduction  measures  (including  those  prior  to  source  selection); 

•  contract  type; 

•  technical  maturity  of  subsystems  and  components; 

•  requirements  growth  (or  not); 

•  “complexity”  and  “density”;  and 

•  funding  instability  (or  not). 

Worth  noting  is  that  some  (perhaps  all)  items  on  this  list  could  also  apply  to  program 
cost  estimation. 

There’s  nothing  original  here;  the  first  four  items  have  been  publicly  cited  as  lessons 
learned  and  applied  to  the  LRSB  program  (Butler,  2015;  Seligman  et  al.,  2015;  Sweetman, 
201 5d;  Tirpak,  2015).  In  addition,  below,  we  discuss  requirements  growth  in  the  F-35 
program. 

“Complexity”  is  suggested  as  an  explanatory  variable  by  a  particularly  interesting 
comment  by  a  senior  DoD  official:  “Our  complexity  reach  exceeds  our  engineering  grasp.”6 
One  plausible  metric  for  complexity  is  lines  of  code  (virtual  complexity  perhaps).  For 
example,  Hallion  (1990)  reports  64,000  lines  of  code  in  the  F-15A  and  2.4  million  lines  in  the 
F-15E.  Lines  of  code  in  the  F-35  vary  with  source  and  date.  A  2014  CRS  report  estimates  F- 
35  software  as  containing  approximately  29  million  lines  of  code  and  still  growing  (Gertler, 
2014,  p.  14). 

In  addition  to  virtual  complexity,  we  could  consider  “density,”  indicating  physical 
complexity.  Density  is  “how  tightly  systems  and  equipment  are  placed  within  a  hull  structure” 
(Grant,  2008).  There  is  other  interesting  research  on  “density”  as  a  cost  driver  for  warships 
(e.g.,  Terwilliger,  2015). 


5  In  fact,  post-mortem  analysis  of  schedules  arguably  fits  directly  within  the  PARCA  charter. 
(http://www.acq.osd.mil/parca/performance-assessments.shtml). 

6  Ninth  Annual  Acquisition  Research  Symposium,  May  2012,  Monterey,  CA.  Comment  understood  as 
not  for  attribution. 
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Schedule  Estimating  Goals 

We  think  that  something  like  the  current  structure  for  cost  estimates  is  a  useful 
analog  in  thinking  about  a  similar  structure  for  schedule  estimates.  This  is  illustrated  in 
Figure  2. 


Figure  2.  Schedule  Estimation  by  Program  Phase 

(adapted  from  Blanchard  &  Frabrycky,  2006,  Figure  17.9,  p.  595) 

Note.  Source  draft  referred  to  cost  estimation  by  program  phase. 

In  that  vein,  a  comprehensive  schedule  estimating  repertoire  would  include  the 
following: 

•  macro-level,  statistical  methods  to  do  those  estimates  in  the  early  stages  of 
the  program  (ex  ante), 

•  more  specific  methods  to  update  schedule  estimates  during  the  program  (in 
media  res),  and 

•  methods  for  explaining  the  results  of  events  and  decisions  previously  in  the 
program  (ex  post). 

For  estimates  done  early  in  the  program,  we  think  “Schedule  Estimating 
Relationships”  featuring  historically  important  schedule  drivers  are  promising.  They  can 
provide  preliminary  estimates  of  acquisition  schedules  to  inform  concept  and  requirements 
determination.  They  could  also  serve  as  an  independent  check  of  scheduling  aspects  of 
bidders’  proposals. 

During  program  execution,  it’s  highly  desirable  for  program  managers  to  have  the 
means  to  update  schedule  estimates.  To  a  considerable  extent  these  already  exist,  as 
discussed,  for  example,  in  the  DAU’s  Scheduling  Guide  for  Program  Managers  (DSMC, 
2001 ).  A  number  of  tools  (discussed  above)  are  available  to  program  managers  and  their 
staffs  (Blanchard  &  Fabrycky,  2006,  esp.  Chaps.  11  &  18). 
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Finally,  schedule  analysis  and  prediction  methods  can  usefully  support  after-the-fact 
(ex  post)  analyses  of  program  successes  and  difficulties.  Such  tools  could  enable  schedule 
analyses  similar  to  those  now  conducted  by  the  OSD’s  Root  Cause  Analysis  office  for 
selected  programs  with  cost  problems. 

Cost-Performance-Schedule  Trade-Offs 

Schedules  arise  from  trades  (perhaps  implicit)  among  cost,  schedule  and 
performance.  And  “making  ...  effective  cost,  performance,  schedule,  and  quantity  trade-offs” 
(emphasis  added)  is  a  major  theme  of  the  JCS  directive  for  the  Joint  Capabilities  Integration 
and  Development  System  (JCIDS;  CJCS,  2015). 

We  know  a  fair  amount  about  the  structure  of  cost,  performance,  and  schedule  trade¬ 
offs,  but  there’s  more  worth  finding  out.  As  one  report  put  it,  “the  literature  linking  cost, 
performance,  and  schedule  is  by  no  means  abundant.  This  is  due  in  large  part  to  the  sheer 
complexity  of  the  interrelations  between  performance  characteristics  and  technical 
specifications,  as  well  as  the  unique  missions  ...  systems”  (Voltz,  1992,  p.  13).  In  this 
section,  we  offer  a  preliminary  explanation  of  those  trade-offs  taken  two  at  a  time:  Cost  and 
Performance;  Cost  and  Schedule;  Performance  and  Schedule. 

Cosf  and  Performance 

Of  these  three,  we  probably  know  most  about  Cost  vs.  Performance.  Basically,  we 
expect  to  pay  more  to  acquire  higher  performance.  Figure  3  shows  a  notional  trade-off  with 
effects  of  technical  progress. 

That  relationship  has  been  investigated  in  a  number  of  empirical  studies.  One  of 
those  (Hildebrandt  &  Sze,  1986,  p.  15)  led  to  the  following  cost-performance  relationship  (in 
log-log  form)  shown  below.  This  is  the  result  of  a  regression  analysis  of  a  data  base  of  66 
fighter  and  attack  aircraft  with  first  flights  from  1950  (F-89)  to  1979  (F/A-18). 

InCAC  =  1 .99  +  ln*P  +  1 .31  ln*ASP  -  .31  InR  -  .03T  -  .50*ATTACK  -  ,89*M0D  +  bj*lnY,  (1 ) 

where  CAC  is  cumulative  average  cost;  P  is  resource  price  levels  (primarily  labor 
and  materials);  ASP  is  an  aircraft  performance  index;  R  is  production  rate;  T  is  year  of  first 
flight;  ATTACK  is  dummy  variable  (1  for  attack  aircraft,  0  otherwise);  MOD  is  a  dummy 
variable  for  aircraft  models  that  are  modifications  or  upgrades  of  an  existing  aircraft  type;  bi 
is  the  relevant  learning  curve  parameter;  and  Y  is  cumulative  production.  Franck  (1992) 
used  the  same  data  to  infer  patterns  of  cost-performance  design  choices. 
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Figure  3.  Cost  vs.  Performance  Trade-Off 

(adapted  from  Gansler,  1987,  p.  K-8;  Sullivan,  1981) 

The  first-flight  variable  is  intended  to  capture  the  effects  of  technical  progress.  All 
other  things  equal,  we  expect  to  pay  less  for  a  given  level  of  performance  with 
improvements  in  technology.  This  is  illustrated  in  Figure  3,  which  includes  effects  of 
technical  progress.  As  previously  stated,  as  performance  increases,  so  does  cost.  However 
advancing  technology  shifts  the  cost-performance  curve  down  and  to  the  right  (lessening  to 
cost  of  any  given  level  of  performance). 

Program  Schedule  and  Cost 

Effect  of  development  time  on  program  cost  is  somewhat  ambiguous.  Keeping  a 
team  in  place  longer  means  greater  overhead  expenses  (sometimes  called  the  “standing 
army”  effect).  But  shorter  development  times  can  mean  less  chance  to  develop  technology 
and  sort  among  alternative  approaches  and  incurring  the  costs  associated  with  cascading 
effects  of  wrong  turns. 

These  seem,  in  general,  to  be  countervailing  effects.  Less  time  means  less  overhead 
cost  over  the  life  of  the  program.  More  time  means  better  chances  to  avoid  pitfalls  and 
manage  risk.  In  theory,  the  best  course  of  action  is  reached  by  balancing  increases  in 
overhead  (indirect)  cost  with  direct  program  cost.  This  is  shown  in  Figure  4. 
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Figure  4.  Total  Cost  Analysis  for  Selecting  Optimum  Program  Duration 

(adapted  from  DSMC,  2001,  p.  60;  Zschau,  1969,  pp.  28-30) 

This  sketches  nicely,  but  solving  the  implied  problem  is  more  complicated.  For 
example,  not  all  relevant  costs  are  internal  to  the  acquisition  program  itself.  As  the  DSMC 
Scheduling  Guide  points  out,  “Each  month  added  to  the  development  and  production  of  a 
new  ...  system  tends  to  reduce  by  1  month  the  operational  life  of  the  product”  (DSMC,  2001, 
p.  61 ).  This  suggests  that  monetized  effects  of  fielding  delay  should  be  added  to  total 
costs — a  difficult  task. 

Nonetheless,  those  costs  of  delay  can  be  all  too  real  and  multifaceted,  as  illustrated 
by  the  F-35  program.  The  effects  included  a  projected  shortfall  of  tactical  fighters  in  both  the 
Air  Force  and  Navy  (Tirpak,  2010;  Trimble,  2010).  To  help  bridge  that  gap,  it  was  necessary 
to  keep  the  older  “legacy”  aircraft  in  service  for  longer  than  originally  planned — and 
consequently  spend  more  money  than  originally  planned  to  retard  their  rate  of 
obsolescence.  For  example,  the  U.S.  Air  Force  has  been  obliged  to  devote  considerable 
resources  to  upgrading  its  “legacy”  fourth-generation  systems  and  to  extending  their 
operational  lives  (GAO,  2012).  Overall,  “the  failure  of  the  so-called  fifth-generation  fighters 
...to  arrive  on  time  and  on  cost  has  cascading  effects  throughout  U.S.  and  allied  fighter 
forces”  (Sweetman,  2012). 

Schedule  and  Performance 

A  notional  representation  of  system  performance  vs.  program  time  appears  in  Figure 
5.  The  figure  implies  that  increasing  program  time  allows  for  a  more  considered  approach 
that  permits  better  decisions.  However,  increases  in  indirect  cost  caused  by  a  longer 
program  crowd  out  resources  directly  useful  for  system  development.  And  beyond  some 
point,  the  slope  of  the  Performance  vs.  Time  curve  goes  negative. 
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Figure  5.  Performance  vs.  Development  Time  Trade-Off 

(adapted  from  Zschau,  1969,  pp.  28-30) 


While  certainly  understandable  in  the  abstract,  there  are  some  difficulties  with  this 
trade-off  in  practice.  Among  other  things,  development  programs  that  proceed  with  a  strictly 
fixed  budget  are  very  rare,  if  not  nonexistent.  This  limits  opportunities  to  develop  a  model 
grounded  in  actual  experience. 

The  Concurrency  Issue 

Program  “concurrency”  is  generally  understood  to  involve  beginning  production  prior 
to  completion  of  development  testing  (DoD,  2015,  p.  46),  or  more  broadly  as  “combining  or 
overlapping  phases”  (“Concurrency,”  n.d.). 

Concurrency  is  frequently  cited  as  a  “high  risk  strategy  that  often  results  in 
performance  shortfalls,  unexpected  cost  increases,  schedule  delays,  and  test  problems” 
(GAO,  2012).  On  the  other  hand,  Goure  (2015)  noted  “a  number  of  reasons  to  pursue 
concurrency,”  including  early  identification  of  production  problems  and  faster  fielding  of  new 
hardware. 

One  very  interesting  study  suggests  an  optimum  level  of  concurrency  (from  the 
perspective  of  cost).  However,  the  authors  did  not  find  strong  empirical  evidence  to  support 
that  hypothesis  (Birchler  et  al.,  201 1 ,  p.  252).  Nonetheless,  some  form  of  dynamic, 
simultaneous-equation  model  might  prove  useful. 

Measuring  Performance 

Metrics  for  cost  and  schedule  time  are  generally  well  understand.  Metrics  for 
performance  are  much  less  definite.  Generally  system  “performance”  is  reported  as  a  vector. 
For  tactical  fighters,  the  elements  of  the  vector  are  characteristics  such  as  maximum  speed, 
service  ceiling,  thrust-to-weight  ratio,  combat  range,  weapons  carriage,  and  Radar  Cross 
Section  (RCS).  One  noteworthy  effort  to  develop  performance  indices  (scalar  measures)  for 
a  variety  of  combat  system  types  was  undertaken  by  the  Analytic  Sciences  Corporation 
(ANSER).  This  occurred  mostly  in  the  1980s  and  described  as  the  TASCFORM  method 
(Regan  &  Voigt,  1988). 

Within  that  overall  project,  the  TASCFORM-Air  model  of  combat  capability  was 
intended  to  assess  tactical  fighters,  attack  helicopters,  and  bombers  in  various  conventional 
missions  (Regan  &  Voigt,  1988,  p.  1-1).  Tactical  aircraft  were  assessed  in  the  context  air-to- 
air  (“air  combat”)  and  “surface  attack”  against  both  land  and  maritime  targets  (p.  2-2).  The 
basic  intent  of  TASCFORM-Air  was  to  systematize  observable  technical  features  and 
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combine  those  with  judgments  of  air  combat  experts  to  provide  (scalar)  indices  of  fighter 
capability  in  several  operational  contexts. 

The  capability  measures  applied  directly  to  individual  aircraft  are  organized  in  a 
hierarchy: 

•  Weapon  Performance  (WP,  a  function  of  weapons  carriage,  range, 
maneuverability  and  speed); 

•  Weapon  System  Performance  (WSP,  WP  plus  target  acquisition, 
susceptibility  to  countermeasures,  weapon  enhancements,  navigation  and 
survivability) 

•  Adjusted  Weapon  System  Performance  (AWSP,  WSP  plus  “obsolescence” 
and  sortie  rate;  p.  2-4). 

So,  how  does  the  F-35  performance  look  relative  to  fourth-generation  fighters?  A 
comparison  of  the  aircraft  types  in  the  Weapons  Performance  dimensions  (which  emphasize 
payload,  range,  maneuverability,  and  speed)  shows  there’s  not  much  difference. 

•  Hard  points:  F-35  has  a  comparable  number  of  weapons  hard  points  relative 
to  the  F-18  and  F-16  but  much  fewer  in  stealth  mode. 

•  Max  Speed:  all  three  aircraft  are  all  comparable. 

•  Ferry  Range  is  comparable,  if  F-35  has  external  tanks. 

•  There’s  a  Combat  Range  advantage  for  the  F-35  when  operating  in  a  high- 
high-high  profile,  compared  with  range  in  a  high-low-high  profile  for  the 
fourth-generation  fighters. 

•  Maneuverability:  Thrust-to-weight  ratio,  max  Gs,  and  wing  loadings  are 
comparable  for  F-16,  F-18,  and  F-35. 

•  Sortie  Rate:  not  yet  determined.  The  F-35  is  still  maturing. 

•  Survivability:  favors  stealthy  aircraft,  but  nonetheless  subject  to 
countermeasures  (discussed  above). 

Force  capability  is  generally  presented  as  numbers  and  types  of  systems.  Force 
capability  indices  are  also  discussed  in  TASCFORM  (pp.  2-4,  2-30-2-36),  in  which  force 
capability  is  assumed  to  be  the  sum  of  individual  performance  (by  tail  number).  These 
measures  do  not  fully  address  force  effectiveness  as  a  function  of  networking  and  shared 
situational  awareness. 

The  fifth-generation  fighter  advocates  have  a  new  perspective  on  system  and  force 
capabilities.  New  aircraft  models  such  as  the  F-22  and  F-35  are  seen  as  disruptive 
innovations.  Within  this  perspective,  the  operational  capabilities  of  the  fifth  generation  are 
due  to  the  combination  (synergy  perhaps)  of  airframe  characteristics  and  “ability  to  work 
within  and  interact  with  a  broad  array  of  networked  systems”  (Deptula,  201 1 ;  Space  Daily 
Staff,  2006). 

Moreover,  fifth-generation  characteristics,  especially  stealth,  increase  the  proportion 
of  resources  devoted  to  offensive  air  operations.  Fifth-generation  aircraft  likely  need  fewer 
fighter  sorties  to  support  penetration  of  advanced  and  integrated  air  defenses,  and  fewer 
tanker  sorties  (due  to  smaller  strike  packages). 

Regardless  of  one’s  opinion  of  fifth-generation  performance  advantages,  it’s  hard  to 
avoid  the  conclusion  that  a  credible  method  of  measuring  system  (and  force)  performance 
should  account  for  the  advantages  of  stealth,  shared  battlefield  awareness,  and  networked 
operations. 
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F-35:  From  CALF  TO  JSF7 

The  Lockheed  Martin  F-35  Joint  Strike  Fighter  (JSF,  Lightning  II)  is  a  single  seat, 
single  engine,  fifth-generation  multirole  fighter  designed  to  perform  ground  attack, 
reconnaisance,  and  air  defense  missions  while  in  stealthy  operation.  It  was  originally 
visualized  as  a  relatively  affordable  strike  fighter  available  in  three  largely  common  versions 
for  the  Air  Force,  Marines,  and  Navy.  It  didn’t  work  out  that  way. 

Then-Major  General  Christopher  Bogdan  (JSF  Program  Executive  Officer  designate) 
commented  that  the  F-35  “is  not  a  single  program  (but  rather)  three  separate  airplane 
programs  (with)  common  avionics  and  a  common  engine.”  He  also  stressed  the  difficulties 
involved  in  reaching  agreement  on  decision-making.  In  his  words,  “It's  hard  enough  to  get 
one  service  to  answer  questions  about  requirements.  Imagine  three  services,  eight  partners, 
and  two  FMS  customers”  (Bogdan,  2012). 

All  three  models  are  designed  for  limited  supersonic  operation,  and  to  carry  their 
primary  weapons  internally,  to  preserve  their  stealth  characteristics.  Although  physical 
differences  arose  from  methods  of  takeoff  and  landing,  requirements  were  also  driven  by 
different  operational  needs. 

The  F-35A  was  a  replacement  for  the  F-16,  the  A-10,  and  perhaps  the  F-1 5  fighter. 

In  addition,  it  is  intended  to  complement  the  F-22  air  superiority  fighter.  The  Air  Force  sought 
an  advanced  attack  aircraft  with  stealth,  advanced  avionics,  and  low  life-cycle  operating 
costs  providing  improved  range,  speed,  and  appreciable  weapons  load  capacity. 

The  F-35B  is  a  short  takeoff  and  vertical  landing  aircraft  (STOVL)  acquired  to  replace 
its  AV-8B  Harrier  and  its  F/A-18A/B/C/D  strike  fighters.  It  was  designed  to  operate  from 
forward  battlefields,  helicopter  carriers,  and  as  a  “jump  jet”  from  smaller  conventional 
carriers.  The  F-35C  (CV)  chosen  by  the  U.S.  Navy  resembled  the  Air  Force’s  F-35A  but  was 
modified  for  carrier  operations.  It  is  intended  to  replace  earlier  versions  of  the  F/A-18. 

Joint  and  International  Nature 

At  the  time  of  JSF  conception,  there  was  a  clear  preference  at  the  highest  levels  of 
the  DoD  for  joint  projects.  Typically,  the  rationale  for  jointness  is  that  a  largely  joint  project 
lessens  costs  of  developing,  procuring,  and  operating  and  supporting  some  large  number  of 
separate  aircraft  designs  with  similar  (but  not  necessarily  identical)  requirements. 

A  study  by  the  RAND  Corporation  undertook  to  examine  this  issue  (Lorell  et  al., 
2013),  which  focused  on  the  costs  of  jointness.  The  critical  finding  is 

the  need  to  accommodate  different  service  requirements  in  a  single  design  or 
common  design  family  leads  to  greater  program  complexity,  increased 
technical  risk,  and  common  functionality  ...  beyond  that  needed  for  some 
variants,  potentially  leading  to  higher  overall  cost,  despite  the  efficiencies  (of 
common  design).  (Lorell  et  al.,  2013,  iii) 


7  This  section  relies  in  part  on  background  information  from  Aboulafia  (2015)  and  Gertler  (2014).  Also 
we  found  the  Wikipedia  article  on  the  F-35  to  be  a  good  source  for  those  seeking  basic  information  on 
the  program  (“Lockheed  Martin  F-35  Lightning  II,”  n.d.). 
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The  F-35  won  the  DoD  source  selection,  with  an  industry  team  of  Lockheed, 

Northrop  Grumman,  BAE  Systems,  and  Pratt  &  Whitney.  (Aboulafia,  2015,  identified  the  F- 
35  suppliers  in  more  detail.) 

From  the  earliest  days  of  the  JSF  project,  the  Office  of  the  Secretary  of  Defense 
stressed  international  participation.  The  UK  joined  the  JSF  project  as  a  Level  1  Full 
Collaborative  Partner.  There  were  five  (Level  II)  Associate  Partners  and  three  (Level  III) 
Informed  Partners  in  the  Systems  Development  and  Demonstration  Phase  (Schreiber,  2002, 
p.  164). 

History  and  Antecedents 

Defense  procurement  funding  fell  sharply  in  the  early  1990s,  implementing  the 
Bottom  Up  Review  recommendations — ending  such  programs  as  the  NATF  (Naval 
Advanced  Tactical  Fighter)  and  the  A-12/ATA.  Fearing  loss  of  domestic  military  aircraft 
design  skills,  the  DoD  undertook  a  series  of  largely  unsuccessful  programs.  This  effort 
include  support  for  design  of  advanced  technology  aircraft  available  for  production. 

The  list  of  aircraft  concepts  not  leading  to  production  includes  the  following  (e.g., 
Aboulafia,  2015,  esp.  pp.  10-11): 

•  A-X/A/F-X,  a  Navy-dominated  joint  program  was  canceled  due  to  the  A-12’s 
high  cost,  and  by  the  1993  appearance  F/A-18E/F  (Super  Hornet). 

•  ASTOVL/SSF  (Advanced  STOVL/STOVL  Strike  Fighter)  was  an  ARPA 
project  intended  to  develop  a  supersonic  AV-8B  Harrier  successor.  NASA 
and  the  UK  both  participated  in  this  effort.  It  was  merged  by  Congress  with 
JAST  in  mid-1994. 

•  CALF  (Common  Affordable  Lightweight  Fighter)  was  the  formal  name  for 
DARPA’s  ASTOVL  project  that  included  a  conventional  take-off  design 
capability.  Sometimes  known  as  the  X-32,  CALF  was  merged  into  JAST  in 
November  1994. 

•  JAF  (Joint  Attack  Fighter)  explored  the  same  ideas  as  JAST,  as  was  also  true 
of  the  JSSA,  the  Joint  Stealth  Strike  Attack  Aircraft. 

•  MRF  (Multi-Role  Fighter)  was  a  Navy/Air  force  program  designed  to  produce 
a  follow-on  aircraft  for  the  F-16,  F/A-18  and  several  other  legacy  planes.  It 
was  sidetracked  by  the  appearance  of  the  F/A-18E/F  (Super  Hornet). 

JAST/ JSF 

The  F-35  Joint  Strike  Fighter  emerged  from  the  Joint  Advanced  Strike  Technology 
Program  (JAST).  However,  JAST’s  original  goal  was  to  develop  technologies  for  advanced 
strike  aircraft  (DSB,  1994).  It  happened  that  JAST’s  plans  to  fund  several  concept 
demonstrator  aircraft  in  1996  coincided  with  ASTOVL’s  planned  timing  of  the  start  of  its 
Phase  III  (full-scale  flight  demonstration).  The  managements  of  both  programs  concluded 
that  it  would  be  logical  to  make  JAST  the  U.S.  military  service  sponsor  for  the  flight 
demonstration  phase  of  ASTOVL.  In  any  case,  FY95  budget  legislation  directed  an 
immediate  merger  of  ASTOVL  into  JAST  (DoD,  JSF  History,  2015). 

In  early  1997,  Lockheed  Martin  and  Boeing  were  selected  to  develop  flying  airframes 
for  the  concept  demonstration  phase.  They  were  designated  X-32  (Boeing)  and  X-35 
(Lockheed  Martin),  respectively,  with  evaluations  between  September  2000  and  August 
2001 .  On  October  26,  2001 ,  the  Lockheed  Martin  team  was  announced  as  the  winner,  after 
which  the  program  transitioned  to  the  JSF  System  Development  and  Demonstration  (SDD) 
phase  (Aboulafia,  2015,  esp.  pp.  11-12). 
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Cost  and  Scheduling  Problems 

Few  new  weapon  systems  have  earned  such  a  widespread  reputation  for  problems 
encountered  in  the  design  and  development  stages  as  the  F-35.  A  few  comments  are  useful 
here.  Early  development  problems  in  many  new  products  were  followed  by  highly  effective 
operational  performance.  The  C-17  is  one  example  (Franck  et  al.,  2012).  But  the  F-35 
involved  much  more  difficult  design  and  development  problems  (Blickstein  et  al.,  2011,  esp. 
pp.  42,  49). 

The  RAND  Corporation  and  others  reviewed  the  Joint  Strike  Fighter  and  provided  a 
root  cause  analysis  of  its  cost  problems.  The  RAND  report  identified  “in  some  measure”  an 
overly  optimistic  government  estimate  of  the  influence  of  acquisition  reform  and 
“produceability  initiatives”  as  responsible  for  underestimates  of  future  procurement  cost 
growth.  When  combined  with  a  perceived  strong  need  for  an  improved  F-16  replacement, 
the  OSD  proved  willing  to  begin  “a  technologically  complex,  highly  concurrent  F-35 
program.”  The  end  results  included  schedule  slippage  and  cost  growth  that  resulted  in  a 
Nunn-McCurdy  breach  (e.g.,  Rutherford,  2010). 

Explaining  the  Time  Curve 

In  this  section,  empirical  models  are  discussed  that  focus  on  the  key  variable: 

Months  from  Initial  Award  to  IOC  (or  Time  to  IOC)  for  fighter  aircraft.  As  shown  previously, 
this  variable  has  increased  with  later  initial  award  years  for  fighter  aircraft. 

In  this  empirical  analysis,  contract-level  data  contained  in  Selected  Acquisition 
Reports  (SARs)  or  the  F/A-18E/F,  F-22,  and  F-35  (and,  where  possible  the  Air  Force  F-35A) 
is  emphasized.  For  each  published  SAR,  generally  on  December  31,  both  program  and 
contract  level  data  are  included.  Because  each  SAR  for  fighter  aircraft  contain  data  for  two 
or  more  contracts  (both  airframe  system  and  engine),  data  at  this  level  not  only  increases 
the  size  of  the  data  set  but  also  permits  inclusion  of  contract  type,  changes  in  the  target 
cost,  years  elapsed  since  time  of  contract  award,  and  contract  variance.  Contract  variance 
information  (not  required  for  Firm  Fixed  Price  contracts)  includes  both  cost  and  schedule 
variance.  The  former  provides  information  on  the  difference  between  the  planned  and  actual 
contract  cost,  the  latter  on  the  difference  between  the  planned  work  performed  and 
scheduled.  (Future  analysis  will  extend  this  work  to  include  program-level  data  including  the 
various  program-level  variances.) 

Examination  of  the  available  data  indicates  that  complex  interactions  among  the 
relevant  variables  complicates  the  traditional  regression  analysis  view  of  explanatory 
variables  affecting  the  dependent  variable.  Our  analysis  includes  both  explanation  and 
association.  Including  association  variables  provides  insight  into  the  strength  of  the 
relationships  between  these  variables  and  the  dependent  variable  (other  variables  held 
constant). 

We  are  also  investigating  professional-judgment  measures  of  fighter  effectiveness 
for  fighters,  which  would  increase  the  regressions’  explanatory  power.  One  variable 
obtained  from  non-SAR  sources,  included  in  the  current  analysis,  is  the  percent  of  an 
aircraft’s  structural  weight  consisting  of  composites. 

To  understand  the  empirical  analysis,  an  influence  diagram  appears  in  Figure  6 — in 
the  form  of  path  analysis  in  which  Time  to  IOC  is  related  to  contract-specific  cost  variance 
and  other  variables.  In  turn,  current  minus  initial  target  cost  is  related  to  contract  variance 
data,  contract  type,  and  several  other  variables.  We  also  identify  the  expected  signs  of  the 
regression  coefficients  when  possible. 
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Broadly  speaking,  Figure  6  displays  those  variables  that  directly  related  to  Months 
from  Initial  Contract  Award  to  IOC,  namely  Contract  Current — Initial  Target  Price,  Program 
Year,  SAR  Year— Contract  Award  Year,  and  Composites  as  a  Percent  of  AC  Structure 
Weight.  There  are  also  indirect  relationships  between  certain  explanatory  variables  and  time 
to  IOC.  This  occurs  through  these  variables’  direct  relationship  with  [Contract  Current — Initial 
Target  Price].  The  variables  with  an  indirect  relationship  with  time  to  IOC  are  Program  Year, 
[SAR  Year— Contract  Award  Year],  [Contract  Cost  Variance  and  Contract  Schedule 
Variance],  Aircraft  MDS,  and  Contract  Type. 

The  only  variables  with  uncertain  sign  of  regression  coefficients  are  Aircraft  MDS  and 
Contract  Type.  It  is  likely  easiest  to  understand  this  diagram  through  a  discussion  of  the 
regression  results.  First,  Figure  6  shows  the  direct  relationship  between  explanatory 
variables  and  Time  to  IOC. 


Figure  6.  Structure  of  the  Regression  Model 

The  results  in  Table  1  show  all  variables  being  statistically  significant  given  the 
hypothesized  signs  of  the  coefficients  in  the  figure.  When  the  current  target  minus  the  initial 
target  price  increases,  this  likely  means  that  a  specification  change  occurred.  One  would 
expect  specification  change  to  be  associated  with  a  longer  Time  to  IOC.  As  Program  Year 
increases,  this  is  likely  related  to  a  longer  program  length.  In  turn,  this  is  likely  related  to  an 
increase  in  Time  to  IOC.  An  increase  in  [SAR  Year— Contract,  Award  Year]  indicates  that  a 
schedule  delay  is  likely. 

The  most  interesting  independent  variable  may  be  [Composites  as  a  Percent  of  AC 
Structural  Weight].  We  show  that  as  this  increases,  which  is  exactly  what  occurred  when 
shifting  from  the  F/A-18E/F  to  the  F-22  to  the  F-35  programs,  the  dependent  variable 
increases.  It  is  known  that  working  with  composite  materials  is  more  complex  than  traditional 
materials,  and  as  result,  can  be  expected  to  increase  the  length  of  the  program  to  IOC. 
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Table  1.  Direct  Effects  for  Months-to-IOC  Regression 


Explanatory  Variable 

Coefficient 

t-statistic 

(Constant) 

50.409 

12.478 

Contract  Current  -  Initial  Target  Price  ($B) 

1.309 

2.880 

Program  Year 

2.030 

10.015 

SAR  Year  -  ContractAward  Year 

0.986 

3.853 

Composites  as  Percent  of  Structural  Weight 

3.051 

21.242 

FFP  Contract 

12.450 

5.920 

Dependent  Variable:  Months  from  Initial  Contract  Award  to  IOC 

R2  =  0.846;  N  =  164 

We  turn  now  to  the  indirect  effects  regression.  We  have  seen  that  [ Contract )  Current 
Target — Initial  Target  Price]  has  a  positive  direct  relationship  with  the  key  dependent 
variable.  But  there  are  also  variables  that  have  a  direct  relationship  with  [Contract!  Current 
Target — Initial  Target  Price],  and,  therefore,  an  indirect  relationship  with  [Months  from  Initial 
Contract  Award  to  IOC].  Table  2  displays  this  set  of  regression  results. 

Table  2.  Indirect  Effects  in  the  Regression  Model 


Explanatory  Variable 

Coefficient 

t-statistic 

(Constant) 

3.222 

3.941 

Cumulative  Contract  Cost  Variance 

-4.810 

-4.725 

Cumulative  Contract  Schedule  Variance 

-6.072 

-3.339 

Program  Year 

-0.213 

-3.934 

SAR  Year  -  Contract  Award  Year 

0.168 

3.506 

F-35 

-1.333 

-3.867 

F/A-1 8E/F 

-1.979 

-4.257 

CPAF  Contract 

-0.860 

-2.019 

Dependent  Variable:  Contract  Current  Target  -  Initial  Target  Price 

R2  =  0.665;  N  =  110;  Financial  Variables,  $B 

The  Cumulative  Contract  Cost  and  Schedule  Variance  coefficients  are  interesting. 
Contract  Cost  Variance  Budgeted  Cost  of  Work  Performed  minus  Actual  Cost  of  Work 
Performed;  and  Schedule  Cost  Variance  equals  Budgeted  Cost  of  Work  Performed  minus 
Budgeted  Cost  of  Work  Scheduled.  So,  as  these  variables  both  increase,  the  extent  to 
which  the  contractor  is  over  budget  and  behind  schedule  decreases.  Therefore,  motivation 
to  revise  target  price  also  decreases,  consistent  with  the  negative  coefficients. 

As  Program  Year  increases,  specifications  become  more  settled,  and  target  price 
revisions  are  less  likely,  consistent  with  the  negative  coefficient.  However,  as  contracts 
become  longer,  requirement  and  specification  changes  become  more  likely. 

One  advantage  of  employing  both  direct  and  indirect  modeling  is  that  one  can  more 
effectively  assess  indirect  effects  of  Aircraft  MDS  and  Contract  Type  on  Time  to  IOC.  Both 
the  F-35  and  the  F/A-1 8E/F  are  negatively  related  to  [Contract!  Current  Target — Initial  Target 
Price],  For  the  F-35,  this  negative  coefficient  offsets  somewhat  the  positive  relationship 
between  [ Contract )  Current  Target — Initial  Target  Price]  and  [Months  from  Initial  Contract 
Award  to  IOC],  Finally,  we  find  that  CPAF  contracts  are  negatively  related  to  [ Contract ) 
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Current  Target — Initial  Target  Price],  likely  meaning  smaller  increase  in  specifications  that,  in 
turn,  increase  target  price. 

Draft  Research  Agenda 

Acquisition  schedules  are  becoming  more  important.  Therefore,  our  final  aim  in  this 
paper  is  to  propose  a  research  agenda  aimed  at  producing  more  accurate  schedule 
estimates,  particularly  in  major  defense  acquisition  programs. 

Schedule  Estimation  Research:  A  Draft  List  of  Questions  and  Tasks 

1 .  What  is  the  current  state  of  schedule  estimation  and  control?  What’s  needed? 
Where  are  the  gaps? 

o  Interview  subject-matter  experts  regarding  current  state  of  schedule 
analysis,  and  areas  for  improvement. 

2.  How  can  operational  performance  metrics  better  capture  contemporary 
operations? 

o  Update  performance  metrics  for  information-age  warfare.  Start  with 
some  existing  method,  such  as  TASCFORM. 

3.  What  model(s)  best  capture  the  trade-offs  among  program  cost  and 
schedule,  as  well  as  operational  capability  of  fielded  equipment?  Can  those 
models  give  insight  into  “troubled  programs,”  with  difficulties  in  cost, 
schedule,  and  performance? 

o  Analyze  previous  case  studies  (e.g.,  from  Kennedy  School  of 
Government)  for  insights  into  program  schedule  drivers. 

o  Publish  new  case  studies  dealing  with  contemporary  acquisition 
programs,  based,  among  other  things,  on  a  thorough  analysis  of 
relevant  SARs. 

4.  What  estimating  relationships  best  capture  time  to  field  new  hardware?  What 
schedule  drivers  are  generally  most  important? 

o  Based  on  available  data,  formulate  and  empirically  test  models  with 
hypothesized  schedule  drivers. 

o  Formulate  and  test  prediction  markets  for  cost  and  schedule 
problems. 

5.  Is  there  a  prediction  market  design  that  would  produce  useful  information 
about  impending  cost  and  schedule  difficulties? 

o  Design  a  prediction  market  for  defense  acquisition  programs.  Test  it  in 
an  experimental  setting. 

While  this  is  a  very  ambitious  research  program,  it  is  readily  decomposable  into 
smaller  projects.  And  that  we  were  able  to  significantly  advance  the  cost  estimating  state  of 
the  art  suggests  we  can  do  the  same  with  schedule  estimation.  Moreover,  we’d  likely  find 
considerable  insights  from  cost  estimation  methods  useful  for  schedule  estimation. 
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Abstract 

We  studied  the  affordability  constraints  placed  on  acquisition  programs  since  Better  Buying 
Power  was  introduced  by  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics  in  2010.  This  initiative  can  be  thought  of  as  extending  programming  from  five  years 
in  the  future  to  the  full  life  of  each  acquisition  program — typically  in  excess  of  25  years — and 
discussing  the  full  plan  at  Defense  Acquisition  Board  (DAB)  meetings.  We  discuss  the 
management  issues  involved  in  carrying  out  this  initiative,  along  with  the  results  it  has  had. 
The  most  significant  outcome  is  that  it  has  brought  Service  programmers  to  the  Office  of  the 
Secretary  of  Defense’s  DAB  process.  Program  managers  now  need  to  have  their  long-term 
plans  approved  by  the  programmers  who  verify  that  they  fit  with  the  long-term  plans  of  the 
Service.  While  an  Affordability  Analysis  is  not  a  cost  estimate,  it  cannot  be  any  more  precise 
than  the  numerous  program  cost  estimates  used  to  conduct  the  analysis. 

Introduction 

The  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
(USD[AT&L])  affordability  initiative  formally  began  in  2010  as  part  of  Better  Buying  Power 
(BBP)  and  has  been  in  place,  with  some  modifications,  ever  since.  Each  Major  Defense 
Acquisition  Program  (MDAP)  and  Major  Automated  Information  System  (MAIS)  program  that 
is  reviewed  by  the  Defense  Acquisition  Board  (DAB)  is  required  to  conduct  an  Affordability 
Analysis  and  present  the  results.  The  Acquisition  Decision  Memorandum  (ADM)  following 
the  DAB  reflects  the  analysis  by  placing  affordability  constraints  on  the  program,  which  will 
be  tracked  to  verify  that  the  long-term  spending  plans  of  the  Service  remain  affordable. 
Affordability  Analysis  was  formally  mandated  in  the  latest  version  of  Department  of  Defense 
Instruction  (DoDI)  5000.02  in  January  2015. 

Affordability  Analysis  is  an  exercise  in  which  the  entire  spending  of  the  Service  is 
projected  over  the  lifetime  of  the  program  in  question,  usually  in  excess  of  25  years.  All 
other  projected  spending  in  the  Service  should  leave  space  for  the  program  in  question 
under  the  expected  top  line,  and  the  purpose  of  the  analysis  is  to  measure  that  space.  Once 
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that  space  is  determined,  many  assumptions  are  made  to  generate  two  simple  constraints: 
one  for  investment  spending  and  another  for  Operations  and  Support  (O&S).  Since  2013, 
the  responsibility  for  this  analysis  has  belonged  to  the  Service  staffs.  Generally,  they  present 
a  “sand  chart”  that  piles  all  spending  by  portfolios  on  top  of  each  other,  adding  up  to  the 
expected  Service  topline,  and  a  second  sand  chart  that  shows  the  expected  spending  for  all 
of  the  programs  in  the  portfolio  that  includes  the  program  under  consideration. 

In  2009,  many  programs  were  ended  early,  including  the  Army’s  Future  Combat 
Systems  (FCS),  the  Marine  Corps’  new  presidential  helicopter,  and  the  Air  Force’s  F-22 
Raptor — of  these,  only  the  F-22  entered  service  at  all.  The  Honorable  Ashton  Carter  was 
then  the  USD(AT&L)  and  the  Honorable  Frank  Kendall  III  was  his  principal  deputy.  Carter 
went  on  to  become  Deputy  Secretary  of  Defense  in  October  201 1  and  then  Secretary  of 
Defense  in  February  2015.  Upon  Carter’s  first  promotion,  Kendall  became  acting 
USD(AT&L)  and  was  confirmed  in  May  2012,  where  he  is  today.  These  two  men  were  the 
original  proponents  of  BBP,  the  first  edition  (1 .0)  signed  by  Carter  and  the  subsequent  ones 
by  Kendall.  The  BBP  initiatives  have  had  the  backing  of  the  same  senior  defense  team  for 
seven  years,  providing  unusual  leadership  continuity. 

The  stated  reason  for  BBP  1 .0  was  to  reduce  spending  by  improving  efficiency.  An 
additional  reason  was  the  idea  that  future  rounds  of  cancellations  like  they  had  just 
experienced  should  not  be  repeated,  and  Affordability  Analysis  would  help  prevent  it. 

In  this  paper,  we  look  at  what  has  happened  in  the  years  since  the  DoD  began 
mandating  Affordability  Analysis.  So  far,  although  a  few  programs  have  been  cancelled, 
another  wave  like  2009’s  has  not  occurred,  although  another  wave  so  soon  would  have 
been  quite  unexpected,  regardless  of  the  policy  that  was  followed.  There  have  been  some 
other  ramifications,  and  they  are  the  subject  of  this  report. 

An  ongoing  tension  exists  within  the  DoD  between  programmers  and  the  acquisition 
community,  and  Affordability  Analysis  is  in  the  center  of  it.  Programmers  consider  all 
spending  over  several  years  and  make  all  of  the  pieces  fit  under  the  assigned  top  line  in  a 
process  repeated  annually.  The  USD(AT&L),  as  the  chief  acquirer,  makes  decisions  about 
programs  individually  as  they  come  up  sporadically  throughout  the  year.  The  USD  wants  to 
prevent  having  portfolios  short  on  funds  because  that  leads  to  stretches  and  cancellations, 
but  his  tools  are  decisions  for  individual  programs.  Affordability  Analysis  is  an  attempt  by  the 
USD  to  solve  this  problem  with  his  tools. 

Most  of  the  research  for  this  paper  was  conducted  using  Acquisition  Decision 
Memoranda,  handouts  presented  at  DABs  by  program  managers,  and  data  archived  in  the 
Defense  Acquisition  Management  Information  Retrieval  (DAMIR)  System;  all  are  marked 
“For  Official  Use  Only  (FOUO).”  Consequently,  there  are  very  few  actual  data  in  this  report. 
We  do  have  a  larger  report  that  includes  all  of  the  data  and,  as  of  this  writing,  the  distribution 
rules  on  it  have  not  yet  been  set. 

The  Goals  of  Affordability  (and  Their  Evolution) 

Reducing  Spending 

The  original  Memorandum  for  Acquisition  Professionals,  Better  Buying  Power: 
Guidance  for  Obtaining  Greater  Efficiency  and  Productivity  in  Defense  Spending,  dated 
September  14,  2010,  was  signed  by  Carter  and  came  to  be  known  as  BBP  1.0.  This  section 
begins  with  a  discussion  of  the  vision  for  affordability  expressed  in  the  original  memo.  It  is 
followed  by  a  more  lengthy  description  of  the  specific  guidance  therein,  with  emphasis  on 
the  establishment  of  affordability  targets  and  requirements  (later  changed  to  affordability 
goals  and  caps). 
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The  2010  Guidance:  BBP  1.0 

BBP  1 .0  presented  a  list  of  23  principal  actions  to  improve  efficiency  in  the  Defense 
acquisition  process.  The  first  five  of  these  actions  are  associated  with  the  “Target 
Affordability  and  Control  Cost  Growth”  area.  The  motivation  is  stated  in  the  first  paragraph  of 
the  17-page  memo  itself: 

To  put  it  bluntly:  we  have  a  continuing  responsibility  to  procure  the  critical 
goods  and  services  our  forces  need  in  the  years  ahead,  but  we  will  not  have 
the  ever-increasing  budgets  to  pay  for  them.  We  must  therefore  strive  to 
achieve  what  economists  call  productivity  growth:  in  simple  terms,  to  DO 
MORE  WITHOUT  MORE.  ...  Secretary  Gates  has  directed  the  Department  to 
pursue  a  wide-ranging  Efficiencies  Initiative,  of  which  this  Guidance  is  a 
central  part.  This  Guidance  affects  the  approximately  $400  billion  of  the  $700 
billion  defense  budget  that  is  spent  annually  on  contracts  for  goods  ...  and 
services.  ...  We  estimate  that  the  efficiencies  targeted  by  this  Guidance  can 
make  a  significant  contribution  to  achieving  the  $100  billion  redirection  of 
defense  budget  dollars  from  unproductive  to  more  productive  purposes  that  is 
sought ...  over  the  next  five  years.  (USD[AT&L],  2010,  p.  1) 

We  can  offer  some  initial  observations  based  on  this  guidance.  The  first  is  that  there 
is  no  statement  of  a  formal  intention  to  “revolutionize”  defense  acquisition;  the  goal  is  simply 
to  achieve  a  specific  amount  of  cost  savings  over  five  years  that  can  be  used  elsewhere 
within  the  Department.  How  these  savings  or  “redirections”  are  to  be  measured  is  left 
unstated.  A  second  observation,  which  is  modified  elsewhere  in  this  and  later  memos,  is  that 
in  the  fundamental  acquisition  tradeoff  between  cost  and  requirements,  neither  is  to  be 
favored  (or  sacrificed);  instead,  these  redirections  are  to  be  achieved  through  improved 
efficiency — presumably  through  better  management  and  oversight. 

The  body  of  the  BBP  1 .0  memo  goes  on  to  direct  23  specific  actions,  broken  into  five 
major  areas: 

•  Target  Affordability  and  Control  Cost  Growth 

•  Incentivize  Productivity  and  Innovation  in  Industry 

•  Promote  Real  Competition 

•  Improve  Tradecraft  in  Services  Acquisition 

•  Reduce  Non-Productive  Processes  and  Bureaucracy 

The  first  of  these  five,  “Target  Affordability  and  Control  Cost  Growth,”  addresses  the 
principal  subject  of  this  paper:  affordability.  The  other  major  areas  will  not  be  discussed  in 
this  document. 

Affordability  Vision,  Circa  2010 

We  begin  with  the  question,  “What  problem  is  the  affordability  approach  of  BBP  1 .0 
intended  to  address?”  This  question  is  not  to  be  asked  in  a  vacuum;  it  depends  on  how  the 
specific  goals  of  affordability  (as  expressed  in  BBP  1 .0)  differ  from  other  policies  and 
oversight  mechanisms  such  as  Nunn-McCurdy  (N-M)  thresholds.  The  memo  offers  the 
following  definition:  “Affordability  means  conducting  a  program  at  a  cost  constrained  by  the 
maximum  resources  the  Department  can  allocate  for  that  capability.” 

One  proximate  cause  that  led  to  BBP  1 .0  was  the  cancellation  of  a  number  of 
programs  after  years  of  development  and  billions  of  dollars  expended;  chief  among  these 
was  the  Army’s  Future  Combat  Systems  (FCS).  The  perception  at  the  highest  levels  of  the 
Office  of  the  Secretary  of  Defense  (OSD),  and  within  the  legislative  and  executive  branches 
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of  the  federal  government,  was  that  FCS  in  particular  had  been  “unaffordable  from  the  start” 
and  that  this  was  widely  known  even  at  program  inception.  The  cancellation  of  this  program 
was  an  embarrassment  to  the  Army  and  to  the  DoD  as  a  whole.  When  FCS  was  a  going 
concern,  no  Affordability  Analysis  was  conducted,  and  it  is  conceivable  that  the  Army  might 
have  made  it  fit.  However,  Tate  et  al.  (2007)  documented  that  the  costs  of  FCS  would  be  far 
higher  than  was  in  the  Army’s  plan.  So,  even  if  the  official  cost  estimate  might  have  made  it 
look  affordable,  the  better  estimate  would  have  made  it  more  difficult  to  fit  in  the  plan. 

The  vision  of  affordability,  then — in  the  context  of  BBP  1 .0 — is  at  least  in  part  to 
“prevent  future  FCSs.”  The  unaffordability  of  FCS  seems  clear  in  hindsight,  but  how  does 
one  tell  which  programs  that  are  currently  being  initiated  are  likely  to  become  “future  FCSs?” 

In  general  terms,  two  concepts  arose  as  part  of  the  vision.  The  first  was  that  the  five- 
year  planning  horizon  associated  with  the  Future  Years  Defense  Program  (FYDP)  was 
insufficient  to  prevent  initiation  of  doomed  programs:  five  years  does  not,  in  general,  even 
cover  the  development  phase  of  large  programs.  Since  most  of  the  program  costs  are 
incurred  during  the  Procurement  and  O&S  phases,  the  costs  of  these  phases  must  be 
explicitly  considered  from  inception  and  not  pushed  off  into  an  out-year  “bow  wave.”  Key 
parts  of  the  guidance,  therefore,  directed  those  responsible  for  managing  the  programs  to 
consider  the  entire  life  cycle  of  the  program — 30  or  40  years — rather  than  “just”  the  FYDP. 

The  second  concept  was  that  programs  should  not  be  considered  in  isolation,  that  it 
must  be  recognized  and  acknowledged  that,  in  constrained  budget  environments,  cost 
growth  in  one  program  will  affect  the  funding  available  for  other  programs.  This,  it  was 
argued,  must  be  formally  recognized  and  tied  to  the  question  that  the  program  manager 
(PM),  the  Service,  and  the  OSD  should  all  have  in  mind:  At  what  point  does  the  cost  of  a 
program  (including  the  opportunity  cost  of  other  systems)  exceed  its  value  to  the  warfighters 
and  taxpayers?  Complicating  matters  is  the  well-known  but  widely  disliked  practice  of 
stretching  out  the  schedule  of  troubled  programs — as  well  as  programs  that  are  not  troubled, 
but  that  must  contend  with  others  that  are.  This  lowers  the  per-year  costs  of  each  of  these 
programs — this  is  the  purpose  of  the  practice — but  generally  increases  the  total  costs  and 
delays  operational  availability. 

BBP  1.0’s  Guidance 

BBP  1 .0  has  five  “principal  actions”  related  to  the  “Target  Affordability  and  Control 
Cost  Growth”  area: 

•  Mandate  affordability  as  a  requirement. 

•  Drive  productivity  growth  through  Will  Cost/Should  Cost  management. 

•  Eliminate  redundancy  within  warfighter  portfolios. 

•  Make  production  rates  economical  and  hold  them  stable. 

•  Set  shorter  program  timelines  and  manage  to  them. 

The  principal  action  mandating  affordability  gave  rise  to  this  paper,  and  we  will  look 
at  it  in  depth.  The  other  four  mostly  are  seen  as  techniques  for  increasing  productivity.  We 
will  also  look  at  “Eliminate  Redundancy  Within  Warfighter  Portfolios”  because  it  is  the  first 
mention  of  portfolios  and  is  necessary  for  understanding  how  Affordability  Analysis  is 
conducted.  BBP  1.0  also  says, 

Requirements  and  technology  level  for  the  [program]  will  have  to  fit  this 
schedule,  not  the  other  way  around.  When  requirements  and  proposed 
schedules  are  inconsistent,  I  will  work  on  an  expedited  basis  with  the 
Services  and  the  Joint  Staff  to  modify  the  requirements  as  needed  before 
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granting  authority  for  the  program  to  proceed.  (USD[AT&L],  2010,  p.  4) 
[Emphasis  in  original,  and  in  all  cases  that  follow] 

This  is  not  a  focus  on  making  certain  that  our  warfighters  have  the  best  stuff 
possible,  but  rather  trading  that  away  to  stay  on  schedule.  Trading  away  requirements 
supports  the  central  mission  of  BBP  1.0:  reducing  spending. 

Mandate  Affordability  as  a  Requirement 

After  presenting  the  definition  of  affordability  given  earlier — “conducting  a  program  at 
a  cost  constrained  by  the  maximum  resources  the  Department  can  allocate  for  that 
capability” — this  principal  action  directs  program  managers  to  “treat  affordability  as  a 
requirement  before  milestone  authority  [will  be  granted].”  The  memo  continues: 

Specifically,  at  Milestone  A,  my  Acquisition  Decision  Memorandum  (ADM) 
approving  formal  commencement  of  the  program  will  contain  an  affordability 
target  to  be  treated  by  the  program  manager  (PM)  like  a  Key  Performance 
Parameter  (KPP)  such  as  speed,  power,  or  data  rate — i.e.,  a  design 
parameter  not  to  be  sacrificed  or  compromised  without  my  specific  authority. 
At  Milestone  B,  when  a  system’s  detailed  design  is  begun,  I  will  require 
presentation  of  a  systems  engineering  tradeoff  analysis  showing  how  cost 
varies  as  the  major  design  parameters  and  time  to  complete  are  varied.  . . . 
This  analysis  would  then  form  the  basis  of  the  “Affordability  Requirement”  that 
would  be  part  of  the  ADM  decision.  ...  this  guidance  will  apply  to  both 
elements  of  a  program’s  life  cycle  cost — the  acquisition  cost  (typically  30 
percent)  and  the  operating  and  support  cost  (typically  70  percent).  For 
smaller  programs,  the  CAEs  [Component  Acquisition  Executives]1  will  be 
directed  to  do  the  same  at  their  level  of  approval.  (USD[AT&L],  2010,  p.  2) 

The  guidance  officially  states  that  the  PM  must  incorporate  an  affordability  target  as 
a  KPP  at  the  Milestone  A  DAB.  Not  stated  here,  but  implied,  is  that  the  PM  must  also 
incorporate  an  affordability  requirement  as  a  KPP  at  the  Milestone  B  DAB,  and  beyond.2 

The  guidance  does  not  formally  state,  nor  really  even  hint  at,  how  these  affordability 
goals  and  caps  are  to  be  calculated.  Many  different  forms  for  the  constraints  were  used  by 
different  programs  at  DABs,  some  of  which  were  difficult  for  OSD  to  observe,  but  it  has 
become  standard  for  APUC  or  PAUC  to  be  used  to  define  the  constraints  when  the  program 
is  buying  many  units,  and  total  investment  to  be  used  for  programs  in  which  that  is  not  the 
case. 


Generally,  the  stated  affordability  definition — “conducting  a  program  at  a  cost 
constrained  by  the  maximum  resources  the  Department  can  allocate  for  that  capability” — 
requires  that  the  Services  quantify  their  allowable  level  of  expenditures  by  capability  area 
and  fit  all  the  programs  in  that  area  within  that  level.  Since  costs  in  a  capability  area  cover 
many  programs,  tradeoffs  must  be  considered  in  applying  a  cap  to  an  individual  program.  It 
is  difficult  to  answer  the  questions:  At  what  point  does  the  cost  of  (for  example)  a  new 
helicopter  become  so  high  that  you  would  rather  cancel  the  program  and  either  live  with  the 


1  The  Components  are  the  military  Services  and  other  agencies. 

2  The  terms  “target”  and  “affordability  requirement”  were  later  replaced  with  “goal”  and  “cap,” 
respectively. 
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old  ones,  or  start  over?  To  what  extent  would  you  rather  cut  back  other  programs  in  the 
portfolio?  The  idea  of  asking  the  PM  and  the  Service  to  think  about  this  before  contract 
award  is  outstanding — but  the  answer  depends  on  many  factors,  some  of  which  change 
over  time  and  only  some  of  which  are  under  the  PM’s  control. 

The  requirement  to  determine  and  state  affordability  goals  and  caps  is  done  to  act  as 
a  trip-wire  for  cost  growth  sufficient  to  require  a  re-examination  of  Service  priorities  and 
available  resources.  It  thus  overlaps  significantly  with  N-M  reporting.  We  have  no  objection 
to  this;  the  target  audience  is  different,  and  it  could  prove  more  useful. 

Eliminate  Redundancy  Within  Warfighter  Portfolios 

This  action  introduces  two  concepts  that  are  fundamental  to  the  affordability  vision. 
The  memo  text  begins  with  the  example  of  a  program  that  the  Army  decided  to  cancel  (thus 
freeing  up  resources  for  other  Army  programs)  based  on  the  fact  that  its  capabilities  could 
be  met  by  other  systems.  It  reads,  in  part, 

This  was  a  classic  value  decision  that  could  not  have  been  made  by  looking 
at  the  ...  program  in  isolation.  The  Army  had  to  look  at  the  entire  “warfighting 
portfolio” ...  to  see  that  [the  program’s  cancellation]  would  not,  in  fact,  result 
in  a  major  sacrifice  of  military  capability.  ... 

I  intend  to  conduct  similar  portfolio  reviews  at  the  joint  and  Department-wide 
level  with  an  eye  toward  identifying  redundancies.  ...lam  directing  the 
components  to  do  the  same  for  smaller  programs  and  report  the  results. 

This  is  the  first  mention  of  the  term  “portfolio”  in  the  Better  Buying  Power  guidance. 
As  the  concept  of  affordability  evolved,  portfolios  of  families  of  programs  (e.g.,  “tracked 
vehicles”  or  “surface  ships”)  became  central.  The  so-called  “sand  charts”  that  must  be 
presented  in  the  affordability  section  of  each  DAB  review  are  snapshots  of  these  portfolios — 
often  created  by  extending  out  indefinitely  the  spending  in  the  last  year  of  the  FYDP. 

The  significance  of  requiring  portfolio  information  to  be  presented  at  DAB  reviews  is 
not  to  be  underestimated,  and  it  represents  something  new  in  the  standard  OSD  Acquisition 
process.  Up  until  this  time,  the  Milestone  reviews  were  between  one  program  manager  and 
the  appropriate  level  of  acquisition  executive,  typically  USD(AT&L)  for  Acquisition  Category 
(ACAT)  I  programs.  The  requirement  to  discuss  interactions  with  other  programs,  even  if 
superficially,  forces  the  PM  to  engage  with  the  Service  prior  to  Milestone  approval.  It  should 
not  escape  notice  that  a  representative  of  the  Service  programmer’s  office  now  has  a  seat  at 
ACAT  I  Milestone  reviews,  which  was  not  formerly  the  case. 

Expecting  offsets  to  come  from  within  a  single  portfolio  is  less  than  ideal,  but  is  a 
significant  step.  The  ability  to  trade  not  just  within  but  between  portfolios,  and  even  between 
Services,  is  a  major  theme  in  the  book  How  Much  Is  Enough?  (Enthoven  &  Smith,  2006) 
and  ought  to  be.  This  is  especially  so  because  the  portfolios  used  are  almost  always  by 
platform  type.  For  example,  trucks  and  utility  helicopters  are  in  different  Army  portfolios 
(transportation  and  aviation),  and  while  there  are  many  missions  where  neither  could 
replace  the  other,  on  the  margins,  trades  between  them  might  be  the  best  choice.  For  a 
cross-service  example,  the  Army’s  AH-64  Apache  helicopters  perform  similar  missions  to 
the  other  Services’  close  air  support  aircraft. 

The  2013  Guidance:  BBP  2.0 

The  Memorandum  Implementation  Directive  for  Better  Buying  Power  2.0 — Achieving 
Greater  Efficiency  and  Productivity  in  Defense  Spending,  or  BBP  2.0,  which  was  signed  by 
Kendall  as  the  Under  Secretary  on  April  24,  201 3,  incorporates  a  number  of  subtle  changes 
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with  respect  to  BBP  1 .0,  dated  two-and-a-half  years  earlier — some  detailed  changes  and 
some  important  “vision  implementation”  changes.  As  this  is  neither  the  genesis  of 
Affordability  Analysis  nor  current,  we  treat  it  with  less  depth  than  the  other  two.  There  were 
two  important  changes  from  this  iteration  of  BBP. 

BBP  2.0  (USD[AT&L],  2013)  states,  “Constraints  stem  from  long-term  affordability 
planning  and  analysis,  which  is  a  Component  leadership  responsibility.”  Explicitly  giving  the 
setting  of  constraints  to  Component  leadership  was  important.  Now  the  Services  would  have 
ownership  of  the  constraints  as  well  as  the  USD  who  signed  the  ADM,  guaranteeing  that  the 
spending  plan  brought  to  the  DAB  would  be  approved  by  Service  leadership.  Might  this  have 
helped  prevent  the  FCS  debacle? 

Perhaps  the  most  stunning  quote  in  BBP  2.0  (USD[AT&L],  2013)  is  this:  “If 
affordability  caps  are  breached,  costs  must  be  reduced  or  else  program  cancelation  can  be 
expected.”  This  may  have  been  implied  before,  but  in  BBP  2.0,  this  threat  became  explicit. 
Kendall  doubled  down  on  the  importance  of  this  initiative.  With  the  costs  of  breaching  so 
clearly  high,  there  might  now  be  pressure  not  only  on  the  program  office  to  not  breach  the 
constraints,  but  also  on  the  OSD,  which  might  also  feel  compelled  to  not  report  a  breach  to 
prevent  having  to  conduct  such  a  severe  action,  which  might  not  be  warranted.3 

In  the  September-October  2013  issue  of  Defense  AT&L,  Chad  Ohlandt,  a 
researcher  at  RAND  then  serving  on  a  detail  at  the  Acquisition  Policy  Analysis  Center  in 
AT&L,  published  an  article  called  “Dispelling  the  Myths  of  DoD’s  Affordability  Policy.”  The 
five-page  article  lays  out  in  very  broad  terms  what  the  Services  are  supposed  to  do  and 
why.  He  wrote  that  “Affordability  is  all  about  using  that  knowledge  to  avoid  starting  or 
continuing  programs  that  we  cannot  reasonably  expect  to  pay  for  in  the  future.”  The  timing 
of  this  article  suggests  that  there  were  still  questions  within  the  acquisition  community  about 
the  purpose  of  Affordability  Analysis  and  how  to  do  it. 

New  Priority:  Technological  Superiority 

By  2015,  Kendall’s  focus  had  shifted  somewhat.  Using  funds  efficiently  was  still 
important,  but  he  was  also  concerned  about  technological  dominance,  and  said  so  in  BBP 
3.0. 


The  2015  Guidance:  BBP  3.0 

The  memo  Implementation  Directive  for  Better  Buying  Power  3.0 — Achieving 
Dominant  Capabilities  Through  Technical  Excellence  and  Innovation,  henceforth  referred  to 
as  BBP  3.0,  was  signed  by  Kendall  on  April  9,  2015.  While  the  commitment  to  affordability 
remained,  the  tone  changed  significantly. 

As  was  the  case  with  BBP  2.0,  BBP  3.0  (USD[AT&L],  2015)  is  brief — this  time  only  a 
single  page.  It  is  accompanied  by  two  attachments:  a  one-page  Summary  Page,  and  a  33- 
page  attachment  called  “Better  Buying  Power  3.0  Implementation  Guidance.”  We  will  again 
discuss  three  parts  of  this  memo,  although  it  will  be  a  slightly  different  aggregation:  the  one- 
page  memo  itself,  the  one-page  Implementation  Guidance  “Overview,”  and  the  half-page 
section  of  the  Implementation  Guidance  that  specifically  refers  to  affordability. 


3  We  expect  most  parents  recall  making  a  threat  that  had  to  yield  compliance  ...  only  to  find 
themselves  holding  the  pieces  of  a  broken  antique  dish  and  now  having  to  decide  if  they  really  are 
going  to  cancel  the  family  vacation. 
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BBP  3.0  Memo  Body 

In  this  memorandum,  Kendall  writes,  “There  is  more  continuity  than  change  in  Better 
Buying  Power  3.0.  Core  initiatives  focus  on:  ensuring  that  the  programs  we  pursue  are 
affordable.  ...  We  will  continue  all  of  these  efforts.” 

On  one  hand,  all  of  the  guidance  about  the  importance  of  maintaining  long-term 
affordability,  via  requirements  reduction  if  necessary,  still  remains  in  place:  “New  in  Better 
Buying  Power  3.0  is  a  stronger  emphasis  on  innovation,  technical  excellence,  and  the 
quality  of  our  products.”  Here  we  see  the  emphasis  on  innovation,  which  is  likely  to 
discourage  trading  capability  for  affordability.  With  less  trading,  there  might  be  more  cost 
growth.  Furthermore,  an  emphasis  on  innovation  will  lead  to  more  ambitious  programs  that 
are  more  likely  to  yield  cost  growth.  Cost  growth  from  either  source  would  squeeze  other 
programs  and  can  lead  to  unaffordable  portfolios.  On  the  other  hand,  ambitious  programs 
sometimes  fail,  and  if  they  are  canceled  they  can  open  up  affordability  space  as  well. 

BBP  3.0  Implementation  Guidance:  Overview 

The  Overview,  page  1  of  the  Implementation  Guidance,  states, 

The  theme  that  ties  the  content  of  BBP  3.0  together  is  an  overriding  concern 
that  our  technological  superiority  is  at  risk.  Potential  adversaries  are 
challenging  the  U.S.  lead  in  conventional  military  capability  in  ways  not  seen 
since  the  Cold  War.  Our  technological  superiority  is  based  on  the 
effectiveness  of  our  research  and  development  efforts. 

Previously,  the  emphasis  had  been  on  reducing  spending.  This  guidance  is  new. 

BBP  3.0  Implementation  Guidance:  Achieve  Affordable  Programs 

While  there  is  a  new  focus  in  BBP  3.0,  much  of  the  guidance  on  affordability  remains 
the  same.  Perhaps  the  most  important  change  is,  again,  in  tone:  “ACAT  I  programs 
projected  to  exceed  approved  caps  will  undergo  a  Defense  Acquisition  Executive  (DAE) 
review  to  determine  appropriate  corrective  action”  (Implementation  Guidance,  p.  2). The  USD 
has  not  given  up  the  possibility  of  cancelling  programs  that  exceed  their  affordability 
constraints,  but  the  apparent  stakes  have  been  lowered  considerably. 

Formal  Guidance:  DoDI  5000.02 

In  January  2015,  Kendall  signed  DoD  Instruction  (DoDI)  5000.02,  Operation  of  the 
Defense  Acquisition  System.  It  is  consistent  with  BBP  3.0  and  codifies  that  all  of  the 
affordability  work  that  had  been  done  before  is  now  required  along  with  many  other  changes 
to  the  process.  The  new  instruction  has  a  five-page  enclosure  entitled  “Affordability  Analysis 
and  Investment  Constraints,”  which  explains  in  some  detail  how  Affordability  Analysis  should 
be  conducted.  It  also  contains  a  simple  example  of  calculating  a  constraint  for  a  fleet  of 
trucks  when  it  is  assumed  that  the  budget,  inventory,  capability,  and  unit  cost  all  will  be 
constant  for  the  foreseeable  future. 

The  Accomplishments  of  Affordability 

Currently,  a  little  more  than  a  third  of  active  acquisition  programs  have  an 
affordability  constraint,  which  implies  that  a  complete  Affordability  Analysis  was  conducted. 
Those  that  do  not  have  one  are  older  programs  and  therefore  have  not  been  through  a  DAB 
recently.  Most  are  post  Milestone  C  and  have  been  in  production  for  a  while. 

The  most  common  form  of  affordability  constraint  is  a  limit  on  Average  Procurement 
Unit  Cost  (APUC).  The  next  most  common  form  limits  Program  Acquisition  Unit  Cost 
(PAUC).  When  BBP  2.0  was  signed  in  April  2013  (discussed  previously),  responsibility  for 
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conducting  the  Affordability  Analysis  and  creating  constraints  was  explicitly  given  to  the 
Services  (BBP  1 .0  did  not  indicate  who  was  to  be  responsible  for  this).  In  the  early  days, 
affordability  metrics  were  sometimes  based  on  many  other  metrics,  such  as  “unit  recurring 
flyaway  cost”  in  one  specified  year  and  “Average  Ship  Acquisition  Cost.”  Most  programs  now 
have  metrics — discussed  later  in  the  Affordability  Metrics  section — that  can  be  easily 
checked  against  a  value  reported  in  the  annual  Selected  Acquisition  Report  (SAR),  usually 
APUC,  PAUC,  or  total  investment. 

This  brings  us,  finally,  to  the  fundamental  question:  To  what  degree  have  actual 
ACAT  I  programs  adjusted  their  plans  as  a  result  of  the  affordability  initiatives?  The  answer 
is:  probably  “a  bit,”  but  it  is  difficult  to  tell. 

The  obvious  place  to  look  for  the  effect  of  Affordability  Analysis  is  in  requirements 
documents.  We  did  look,  and  found  no  evidence  that  they  were  influenced  by  affordability 
constraints.  We  were  unable  to  find  any  requirements  documents  written  over  the  last  five 
years  in  which  a  requirement  was  relaxed  and  was  clearly  done  to  make  a  program 
affordable.  We  also  did  not  hear  such  stories  from  our  interviews  with  members  of  the 
acquisition  community;  what  we  did  hear  were  accounts  of  programs  that  changed  how  they 
met  requirements  or  bought  hardware.  The  biggest  change  we  noted  was  the  role  of  the 
Service  programmers,  often  called  “the  8s”4  in  the  acquisition  process. 

While  changes  to  constraints  were  fairly  easy  to  find,  changes  to  programs  were 
much  harder,  for  two  reasons.  First,  the  barrier  between  the  Services  and  OSD  precludes 
insight  into  how  the  Services,  and  the  Program  Offices,  have  actually  reacted  to  the 
affordability  guidance  presented  in  BBP  1 .0.  Second,  there  are  many  factors  that  separate 
programs  that  stay  on  track  from  those  that  do  not.  These  include  contractor  competence, 
program  manager  talent,  number  and  magnitude  of  technical  challenges,  stability  of  funding, 
stability  of  requirements,  and  a  variety  of  unknown  unknowns — all  in  addition  to  affordability 
guidance.  It  is  difficult  for  the  OSD  to  sort  out  these  effects. 

Changing  Constraints 

If  constraints  change  too  easily,  then  they  are  not  constraining.  Kendall  has  said  that 
he  will  modify  affordability  constraints  if  there  is  a  change  in  quantity,  so  we  wanted  to  see 
how  often  affordability  constraints  changed.  While  there  is  an  official  list  of  affordability 
constraints  in  DAMIR,  that  file  does  not  include  changes,  only  those  that  are  currently  in 
force.  The  Office  of  the  Secretary  of  Defense  (Acquisition  Resources  and  Analysis)  kindly 
gave  us  a  spreadsheet  that  tracks  all  constraints  ever  levied.  That  file  showed  that  there  are 
17  programs  that  have  had  their  affordability  constraints  changed. 

Of  these  17,  only  four  had  one  binding  cap  changed  for  another,  as  opposed  to  a 
non-binding  goal  replaced  by  either  a  new  goal  or  cap.  The  four  caps  all  changed  on  the 
same  day  in  2015.  One  of  these  four  programs  had  suffered  significant  cost  growth  but  was 
also  deemed  to  be  important  enough  to  warrant  a  higher  cap.  One  ADM  raised  the  cap  for 
that  program  and  lowered  the  caps  for  three  others  in  the  same  Service  portfolio. 


4  “The  8s”  refers  to  the  Army’s  G8,  the  Air  Force’s  A8,  and  the  Navy’s  N8.  Each  of  those  is  an  office 
on  the  Service  staff  that  programs  funds  over  multiple  years.  The  Navy’s  N8  has  delegated  their  role 
at  DABs  to  N2/N6  or  N9  for  most  programs;  these  offices  also  take  a  long  view  of  their  portfolios. 
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We  do  not  know  what  underlying  analysis  went  into  these  new  caps.  The  ADMs  that 
we  read  (all  marked  FOUO  and  therefore  not  publicly  available)  only  show  what  the  new  and 
old  caps  were;  we  could  not  see  if  meeting  the  new  constraints  yielded  a  portfolio  that  was 
just  as  affordable  as  meeting  the  old  constraints,  because  the  constraints  were  in  different 
base  years  and  each  constraint  was  associated  with  a  different  spending  profile.  Our 
experience  suggests  that  these  calculations  were  done  by  a  staff  member  at  the  Service 
and  were  accepted  by  the  OSD  after  some  scrutiny.  Still,  this  clearly  shows  that  the  OSD 
and  the  Service  were  thinking  about  affordability  in  terms  of  a  portfolio  of  programs  and  not 
one  program  at  a  time. 

Bringing  in  the  Service  Programmers 

The  new  affordability  mandates  have  brought  representatives  of  the  Service 
programming  offices  to  the  table  for  Milestone  reviews.  This  has  improved  the 
communication  between  the  programmers  and  the  acquisition  communities  inside  the 
Services.  The  long-term  spending  plans  presented  at  a  DAB  in  the  past  may  not  have  been 
seen  by  the  Service’s  programmer.  Making  the  Services  responsible  for  “owning” 
affordability  forces  the  PMs  and  the  programmers  to  interact  on  these  issues  far  more  than 
they  have  in  the  past. 

Every  year  the  DoD  sends  the  SARs  (prepared  by  program  offices)  and  the 
President’s  Budget  (PB;  prepared  by  the  Service  programmers)  to  the  Congress.  Within  the 
FYDP,  these  must  agree.  However,  for  years  beyond  the  FYDP,  there  can  be  significant 
disagreement  between  what  the  two  documents  say.  For  example,  both  the  F-35  and  Joint 
Light  Tactical  Vehicle  (JLTV)  programs  show  discrepancies  between  the  December  2014 
SAR  submission  and  the  January  2015  PB  submission.  In  the  2016  budget  submission,  the 
Navy  reported  total  costs  for  the  F-35C  carrier  variant  of  $55.66  billion  and  for  the  F-35B 
short  take  off  and  vertical  landing  variant  of  $47.66  billion,  for  a  total  of  $103.32  billion  over 
the  life  of  the  program.  The  December  2014  SAR  lists  the  combined  total  at  $86.8  billion. 
These  numbers  clearly  show  that  even  in  the  era  of  Affordability  Analysis,  the  N8  that  wrote 
the  budget  submission  and  the  program  office  that  wrote  the  SAR  were  not  on  the  same 
page.  Affordability  Analysis  will  not  fix  that  annual  problem,  but  it  does  require  agreement  at 
DABs  when  both  groups  are  in  the  room. 

Affordability  Analysis  also  demands  longer  term  planning  from  the  programmers. 
Before  Affordability  Analysis,  only  the  five  years  in  the  FYDP  received  significant  focus.  Now 
they  are  required  to  plan  over  longer  durations.  The  Army  has  a  new  tool  called  the  Long 
Range  Investment  Requirements  Analysis  (LIRA),  which  they  use  for  this  purpose.  LIRA 
tracks  planned  Army  expenditures  over  many  years,  which  is  exactly  what  Kendall  has 
required.  Unfortunately  for  the  OSD,  the  G8  has  stated  that  the  Army  does  not  intend  to 
grant  access  to  LIRA  to  any  other  organization — this  system  is  for  Army  internal  use  only, 
which  means  that  while  the  OSD  can  look  at  the  results  of  the  Army’s  long-term  studies, 
unlike  with  the  FYDP,  they  will  not  be  able  to  verify  or  validate  the  models  or  their  inputs.  We 
believe  the  other  Services  have  similar  systems  and  similar  concerns  about  sharing  data. 

Ground  Combat  Vehicle 

In  201 1 ,  the  Army’s  Ground  Combat  Vehicle  received  Milestone  A  authority  but  no 
affordability  constraint,  and  it  appeared  in  PB  2014.  But  the  program  went  no  further  in  the 
acquisition  process.  The  vehicle  they  planned  to  buy  was  longer  and  heavier  than  had  been 
anticipated,  which  likely  would  have  presented  significant  operational  difficulties.  However, 
affordability  was  also  a  problem,  as  it  would  have  needed  more  than  half  of  the  expected 
funds  in  the  combat  vehicles  portfolio.  That  this  program  went  no  further  is  a  success  for 
which  Affordability  Analysis  can  claim  at  least  partial  credit. 
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Management  Considerations 

To  make  Affordability  Analysis  as  useful  as  possible,  there  are  several  factors  that 
need  to  be  thought  through.  While  it  has  already  yielded  some  wins  for  the  DoD,  as 
discussed  previously,  we  think  some  improvements  could  be  made.  We  also  want  to 
highlight  what  is  working  well. 

Affordability  and  Cost  Estimates 

The  relationship  between  the  affordability  of  a  program  and  the  cost  estimates  of  the 
programs  in  its  portfolio  should  be  considered.  Affordability  constraints  are  not  cost 
estimates,  and  for  any  program  that  is  going  forward,  the  constraint  must  be  greater  than  or 
equal  to  the  cost  estimate — otherwise  it  ought  not  to  proceed.  However,  what  cost  data 
should  be  used  for  the  other  programs  in  the  affordability  analysis?  A  program  can  become 
unaffordable  because  cost  estimates  have  risen  for  other  programs  in  its  portfolio. 

Consider  new  program  A  which  will  be  in  a  portfolio  with  incumbent  programs  Z,  Y, 
and  X.  Each  incumbent  program  has  a  cost  estimate  that  should  be  in  their  SARs  and 
budget  submissions,  but  also  an  affordability  constraint  that  is  higher.  Should  A’s  target 
assume  that  Z,  Y,  and  X  each  stay  within  their  cost  estimates  or  that  they  float  up  closer  to 
their  affordability  targets?  If  only  cost  estimates  are  used,  programs  could  see  cost  rises  that 
make  the  portfolio  unaffordable  without  any  one  exceeding  its  constraint.  However,  if  the 
affordability  targets  are  assumed,  the  space  for  program  A  is  smaller  and  the  difference 
between  the  cost  estimates  and  the  affordability  constraints  might  be  seen  as  a  “slush  fund” 
to  be  taken  away  from  the  portfolio.  So  far,  it  seems,  the  Services  are  assuming  that  all 
programs  in  the  portfolio  will  stick  to  the  cost  estimates  when  doing  their  affordability 
analyses,  making  it  possible  that  all  programs  could  remain  under  their  constraints  and  still 
yield  an  unaffordable  portfolio. 

Affordability  Metrics 

Affordability  metrics  should  be  designed  so  that  the  USD  can  be  notified  when 
something  is  happening  that  requires  his  attention  but — as  long  as  the  program  does  not 
threaten  the  portfolio’s  affordability — allows  it  to  continue  without  his  involvement. 

Investment  Metrics 

One  natural  way  to  make  an  affordability  constraint  would  be  to  say  that  the  Service 
may  spend  no  more  than  Xy  on  the  program  in  each  year  j  from  the  present  to  the  expected 
end  of  the  program.  This  sequence  of  numbers  is  what  a  detailed  Affordability  Analysis 
yields.  However,  this  has  never  been  used  and  there  are  at  least  two  reasons  this  ought  not 
to  be  adopted.  First,  such  a  requirement  would  take  away  much  discretion  in  future  years. 
There  may  be  good  reasons  to  increase  the  spending  in  one  year  and  decrease  it  in 
another:  perhaps  to  get  the  capability  in  the  field  sooner  or  simply  as  a  trade  to  increase 
efficiency  by  buying  at  a  higher  rate.  Historically,  this  discretion  has  belonged  to  the 
Services,  and  Kendall  has  not  suggested  that  he  wants  to  take  it  away.  Another  reason  not 
to  adopt  this  requirement  is  that  it  is  complicated  to  state.  Kendall  wants  to  describe  the 
affordability  constraint  simply  in  an  ADM,  and  while  he  has  used  tables  with  three  numbers 
for  the  F-35,  this  approach  would  require  a  table  with  as  many  as  40  numbers  in  it,  which 
might  be  unwieldy. 

One  simplification  would  be  maximum  annual  obligations.  The  ADM  would  say,  “This 
program  may  not  exceed  X  dollars  in  any  given  year.”  This  relates  to  affordability;  as  long  as 
the  annual  obligations  stay  low,  other  programs  will  also  be  affordable.  Because  program 
funding  generally  is  not  flat,  in  some  years  the  cap  would  be  higher  than  the  available 
dollars,  but  that  would  be  sorted  out  by  the  Service  programmers.  Unfortunately,  this  metric 
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not  only  allows  stretches  and  increases  to  total  cost,  it  practically  demands  them  when  there 
are  cost  problems.  While  this  does  relate  to  affordability,  it  is  likely  to  be  counterproductive. 

As  discussed  earlier,  AT&L  and  the  Services  have  largely  settled  on  the  use  of  the 
APUC  or  PAUC5  as  the  metric  of  choice  for  most  programs  because  they  can  track  it 
annually  when  the  SARs  are  written.  Also  in  use  are  metrics  based  on  total  investment  or 
total  procurement  dollars.  Typically,  metrics  based  on  totals  are  used  for  programs  such  as 
the  GPS  Operational  Control  System  (OCX)  or  Space  Fence,  where  the  program  is  buying  a 
single  capability — not  some  integer  number  of  identical  (or  more  often  similar)  items  like 
ships,  missiles,  or  ground  vehicles.  Total  expenditure  metrics  are  also  easily  tracked  by  the 
SARs. 


The  primary  problem  with  APUC  and  PAUC  is  that  they  are  not  closely  related  to 
affordability.  If  a  Service  has  a  problem  with  affordability,  they  can  reduce  the  number  of 
units  they  plan  to  buy  or  stretch  the  buy  over  more  years.  Either  choice  will  decrease  the 
costs  in  each  year,  making  the  portfolios  more  affordable.  At  the  same  time,  these  actions 
increase  unit  cost.  While  this  appears  to  be  a  “bug,”  it  is  actually  a  “feature.”  It  means  that 
the  USD  will  be  alerted  and  forced  to  act  when  the  Service  makes  a  decision  that  increases 
unit  costs  in  order  to  make  a  program  fit  in  the  budget. 

A  weakness  of  unit  cost  is  that  even  for  programs  that  are  buying  many  units,  the 
definition  of  “one”  is  not  always  clear.  For  example,  the  Army’s  ATIRCM/CMWS5F6  program 
bought  two  different  systems  for  the  protection  of  helicopters.  Some  “units”  were  only 
CMWS  systems  and  others  included  both.  They  also  had  some  other  accounting  choices 
that  affected  unit  cost  (Balaban  et  al.,  2010).  The  Navy’s  Integrated  Defensive  Electronic 
Countermeasures  (IDECM)  program  is  similar,  with  different  “blocks”  all  included  together. 
Some  of  the  “units”  include  avionics  systems  and  others  include  only  replaceable  decoys.  In 
the  Air  Force’s  Global  Hawk  program,  each  unit  was  a  single  remotely  piloted  aircraft,  so 
counting  units  was  fairly  straightforward,  but  the  prices  varied  significantly  from  one  variant 
to  another  because  the  payloads  were  very  different,  and  some  payloads  were  included  in 
the  Global  Hawk  program  and  others  were  not.  It  is  not  uncommon  for  the  program  office  to 
be  able  to  change  the  mix  of  what  it  plans  to  buy,  which  may  make  the  unit  cost  look 
favorable  even  as  costs  rise. 

Total  expenditure  metrics  are  similar  to  unit  cost,  but  without  the  units  in  the 
denominator.  Stretching  the  program  has  the  same  effect  here  as  it  does  in  unit  costs.  While 
very  few  programs  that  buy  integer  systems  have  used  these,  we  think  more  should 
consider  it.  This  metric  has  the  benefits  of  average  unit  cost  in  that  a  stretch  can  trigger  an 
affordability  breach,  but  it  is  also  more  closely  related  to  affordability.  A  drawback  to  total 


5  APUC  is  the  total  procurement  dollars  in  a  base  year  divided  by  the  total  number  of  production  units. 
PAUC  is  the  total  dollars  in  the  program  (RDT&E  plus  procurement)  divided  by  the  total  number  of 
units.  Both  metrics  are  set  in  Acquisition  Program  Baselines  (APBs)  when  programs  go  through 
milestones.  The  PAUC  and  APUC  are  calculated  each  year  and  compared  to  the  APB  to  determine  if 
there  is  an  N-M  Breach.  Using  them  for  affordability  targets  introduces  another  use  for  these 
numbers.  Each  year,  the  PAUC  or  APUC  is  compared  to  the  affordability  constraint  to  see  if  the 
affordability  constraint  has  been  breached. 

6  The  name  ATIRCM/CMWS  is  a  combination  of  two  systems.  One  system  is  the  Common  Missile 
Warning  System  (CMWS)  and  the  other  is  the  Advanced  Threat  Infrared  Countermeasure  (ATIRCM). 
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expenditure  is  that  programs  that  are  successful  and  have  their  quantities  increased  then 
look  unaffordable. 

Another  interesting  consideration  regarding  choosing  between  total  investments  and 
average  unit  cost  is  in  long-term  plans.  If  the  metric  used  is  average  unit  cost,  program 
offices  are  incentivized  to  show  more  and  more  units  going  out  into  the  future  because  these 
units  can  show  increased  learning,  thereby  lowering  costs,  and  they  provide  more  units  over 
which  to  spread  development  costs.  Total  investment  encourages  programs  to  report  fewer 
units  into  the  future.  Because  the  N-M  rules  already  use  PAUC  and  APUC,  the  combination 
of  the  N-M  rules  and  affordability  rules  would  provide  counter-balancing  incentives. 

O&S  Metrics 

As  the  dominant  life-cycle  cost  of  most  programs,  O&S  costs  are  critical  to 
maintaining  affordability  in  the  broadest  sense. 

Maintenance  practices  have  changed  significantly  in  the  age  of  digital  electronics, 
composite  materials,  parts  obsolescence,  and  technology  refreshes.  We  note  that  the  lone 
example  of  O&S  costs  in  the  January  7,  2015,  version  of  DoDI  5000.02  involves  a  low-tech 
example  of  a  truck  program  (DoD,  2015).  The  problem  of  developing  a  practical 
methodology  for  estimating  O&S  costs  for  a  modern,  high-tech  program  at  inception — that 
is,  Milestone  A — is  larger  than  affordability,  but  the  portfolio’s  affordability  cannot  be 
accurately  estimated  without  estimating  the  O&S  costs. 

To  accurately  model  future  O&S  costs,  one  must  first  be  able  to  accurately  determine 
these  costs  for  current  programs.  Goeller  et  al.  (2014),  along  with  researchers  in  many  other 
organizations,  discovered  that  allocating  O&S  costs  to  programs  is  vastly  more  difficult  than 
assigning  RDT&E  and  Procurement  costs,  although  it  is  improving.  There  are  a  number  of 
reasons  for  this: 


•  Commonly,  the  O&S  resources  of  several  programs  are  combined  into  a 
single  Program  Element,  making  isolation  difficult. 

•  Often  O&S  costs  of  one  system — for  example,  a  cruise  missile — are  actually 
funded  out  of  another  program — for  example,  a  B-52  wing. 

•  The  actual  logs  of  expenditures  are  not  all  centrally  located,  despite  large 
efforts  to  implement  programs  such  as  Visibility  &  Management  of  Operation 
&  Support  Cost  (VAMOSC)  and  Air  Force  Total  Ownership  Cost  (AFTOC). 

•  In  some  cases,  maintenance  is  covered  by  a  warrantee  contract  with  the 
vendor  that  supplied  the  system — meaning  that  the  cost  to  maintain  that 
system  is  not  only  unknown  to  the  government,  it  is  contractor  proprietary. 
This  maintenance  is  funded  with  procurement  dollars  rather  than  Operation  & 
Maintenance  dollars  and  can  be  years  away  from  when  the  maintenance  is 
performed. 

•  Even  where  O&S  costs  can  be  isolated  by  program,  the  funding  often 
represents  what  the  maintenance  organization  was  given — and  not  what  they 
actually  needed  to  satisfy  all  of  their  requirements.  This  problem  can  go  in 
both  directions — a  plane  might  fly  more  hours  than  required  because  they 
have  the  funds  or  it  may  fly  fewer  hours  than  is  considered  optimal  because 
there  were  insufficient  funds  to  support  more.  Actual  O&S  costs  are,  in  fact,  a 
combination  of  what  is  required  and  what  is  provided. 


All  of  these  problems  are  being  worked  on,  and  even  a  casual  look  at  the  SARs 
today  show  that  the  work  here  is  more  sophisticated  and  careful  than  it  was  five  years  ago. 
There  are  other  issues  besides  difficulty. 
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Placing  a  requirement  on  O&S  costs  for  a  program  in  development  could  provide 
poor  incentives  to  the  program  office.  Because  actual  costs  are  likely  to  be  analyzed  even 
on  prototype  hardware,  suboptimal  decisions  about  how  to  operate  and  test  it  might  be 
made.  Perhaps  a  truck  must  be  tested  in  sandy  conditions,  where  it  is  particularly  difficult  to 
maintain.  Because  of  the  high  costs  associated  with  this,  a  PM  might  feel  compelled  to  run 
another  meaningless  long  test  in  more  benign  conditions  to  lower  the  measured  O&S  costs. 

It  is  not  clear  what  the  O&S  constraints  that  have  been  set  will  do,  and  the  way  they 
are  phrased  makes  them  quite  different.  Some  are  totals  over  many  years,  which  would 
provide  different  incentives  than  others  that  are  on  a  per-year  basis,  so  a  program  could 
meet  the  constraint  in  some  years  and  not  in  others.  In  any  event,  once  the  O&S  costs  are 
the  dominant  cost  in  a  program,  the  USD(AT&L)  usually  has  very  little  say  over  the 
program’s  future.  Would  the  Under  Secretary  want  a  new  program  started  to  replace  a 
fielded  system  because  the  O&S  costs  are  too  high?  This  is  unclear.  Designing  systems 
with  an  eye  toward  lower  O&S  down  the  road  is  wise,  but  it  is  not  at  all  clear  what  an 
affordability  constraint  can  accomplish. 

Limits  of  Affordability  Analysis 
Current  Bow  Wave 

The  “bow  wave”  has  been  a  concern  in  the  Pentagon  since  at  least  the  Kennedy 
Administration,  when  Secretary  of  Defense  Robert  McNamara’s  team  created  the  FYDP  to 
extend  planning  horizons.  The  FYDP’s  “out  years”  are  not  a  perfect  prediction  of  the  future, 
but  they  do  enforce  a  level  of  discipline  to  Service  programmers  and  ensure  that  there  is 
some  possible  way  to  continue  five  years  out  with  the  spending  plans  of  today;  there  cannot 
really  be  a  bow  wave  within  five  years,  anymore.  However,  there  can  be  a  bow  wave 
beyond  the  FYDP  that  will  cause  headaches  for  programmers  when  those  bills  come  due; 
Affordability  Analysis  is  intended  to  reduce  that. 

Today,  some  analysts  perceive  a  large  bow  wave  beyond  the  FYDP  in  large  part 
because  of  big  programs  like  the  Navy’s  new  ballistic  missile  submarines  and  the  Air  Force’s 
long  range  strike  bomber  (LRSB;  Gertler,  2015;  Hale,  2016).  In  an  ideal  world,  Affordability 
Analysis  would  make  this  bow  wave  impossible.  These  programs  are  both  in  the  early 
stages,  meaning  there  is  significant  uncertainty,  but  they  are  likely  to  be  expensive.  We  will 
not  assert  that  this  proves  Affordability  Analysis  has  failed,  but  were  Affordability  Analysis 
not  in  use,  the  bow  wave  might  be  worse.  More  than  half  of  all  acquisition  programs  still 
have  no  affordability  constraints.  Affordability  Analysis  is  a  tool  that  may  make  the  bow  wave 
easier  to  deal  with  over  time. 

Affordability  Games 

In  some  ways,  the  acquisition  system  is  a  game,  and  the  laws,  regulations,  and 
policy  are  the  rules.  Affordability  Analysis  and  constraints  are  new  rules,  and  they  have  led 
to  some  gaming  by  the  Services  and  program  offices.  The  JLTV  is  one  case. 

In  Figure  1,  the  horizontal  axis  shows  cumulative  units  delivered.  Each  black  circle 
represents  an  annual  lot  delivery,  and  three  of  them  are  called  out  by  year  to  orient  the 
reader.  The  dots  and  the  solid  red  line  show  what  we  call  the  “Cumulative  Average  Unit 
Cost.”  This  is  what  the  program’s  APUC  would  be  if  the  program  were  executed  until  that 
point  and  then  terminated.  If  the  2015  lot  were  purchased  and  nothing  else,  the  program’s 
APUC  would  be  $835,000.  This  is  normal;  it  is  expected  that  the  longer  the  program  runs, 
the  lower  the  APUC  should  be.  Two  things  about  this  chart  are  particularly  noteworthy.  First, 
according  to  the  black  dots,  the  program  will  not  meet  the  affordability  goal  set  at  Milestone 
B  unless  it  continues  producing  according  to  plan  until  at  least  2038.  Second,  starting  in 
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2028,  for  no  known  reason,  the  cost  estimate  starts  to  fall  below  the  fitted  learning  curve. 
Without  this  unexplained  decrease,  the  JLTV  would  never  meet  its  affordability  target.  The 
chart  may  make  the  differences  look  small,  but  in  2040,  if  the  costs  each  year  match  the 
learning  curve  instead  of  the  prediction,  the  total  extra  cost  would  be  $300  million  over  25 
years. 
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Figure  1.  JLTV  Costs  in  BY  2012  Dollars  Based  on  the  December  2014  SAR 

Estimate 

The  most  recent  PB  submissions  for  JLTV  show  a  significant  decrease  in  cost  with  a 
new  APUC  of  $333,000.  This  is  probably  good  news  for  the  Army  and  taxpayers.  We  do  not 
know  if  the  program  has  achieved  this  by  finding  efficiencies,  reducing  capability,  or  merely 
quantifying  optimism.  This  change  was  not  made  to  satisfy  the  existing  affordability  cap,  as 
they  met  that  the  year  before  and  it  did  not  require  a  change.  It  is  possible  that  the  Army 
conducted  its  own  internal  Affordability  Analysis  and  decided  they  needed  to  reduce  the  cost 
of  this  program.  Whatever  the  case,  it  is  clear  that,  for  a  while,  the  JLTV  program  office  was 
showing  some  strange  numbers,  apparently  to  keep  their  program’s  costs  below  the  cap 
assigned  at  Milestone  B. 

Innovation  and  Predictability 

The  first  two  BBP  memos  were  about  reducing  spending.  This  is  a  laudable  goal,  but 
it  cannot  be  the  DoD’s  only  one.  BBP  3.0’s  full  title  includes  the  words  Achieving  Dominant 
Capabilities  Through  Technical  Excellence  and  Innovation — which  suggests  another  focus  is 
coming  back  to  the  fore:  The  DoD  should  be  acquiring  state-of-the-art  systems.  Designing 
such  systems  is  inherently  difficult  and  unpredictable;  it  is  also  a  long-standing  American 
tradition. 
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Unfortunately,  Affordability  Analysis  is  predicated  on  knowing  costs.  Every  program 
in  the  portfolio  has  a  cost  estimate.  Those  estimates  are  combined  with  the  expected 
budgets  to  determine  how  much  funding  is  available  for  the  system  under  evaluation.  If 
those  cost  estimates  are  highly  uncertain,  it  is  impossible  to  know  how  much  extra  funding  is 
available.  If  any  of  those  programs  are  pushing  the  state  of  the  art,  it  is  difficult  to  know  what 
they  will  cost.  FCS  may  have  gone  too  far,  but  reaches  in  the  past  have  yielded  excellent 
results,  and  we  need  those  from  time  to  time.  We  present  a  historic  system  that  shows  how 
long  this  problem  has  been  around. 

Ian  Toll’s  2008  book,  Six  Frigates:  The  Epic  History  of  the  Founding  of  the  U.S. 

Navy,  tells  of  the  Washington  Administration’s  program  to  build  six  heavy  frigates  as  the 
backbone  of  a  new  navy.  “The  estimated  cost  of  construction,  victualling,  and  three  months’ 
pay  for  officers  and  crew  was  $600,000.  It  was  an  estimate  that  would  seem  preposterous  in 
retrospect.”  This  was  a  huge  sum  at  the  time,  dwarfing  all  federal  expenditures  other  than 
the  interest  on  the  enormous  national  debt  that  had  been  accumulated  during  the  War  of 
Independence,  and  then  there  were  huge  cost  growth  and  schedule  slips  besides. 

The  program  was  plagued  with  many  of  the  issues  we  see  today.  Dramatic 
requirements  changes — is  their  purpose  to  defeat  the  Barbary  Pirates  or  fight  the  navies  of 
France  and  Britain?  Uneven  funding — at  one  point,  the  Congress  required  that  the  program 
be  reduced  from  six  to  three  ships,  but  they  then  changed  their  minds  again.  Pork  barrel 
spending  (before  the  term  was  invented) — the  six  ships  were  built  in  six  cities,  a  decision  Mr. 
Washington  made,  knowing  that  he  was  trading  away  efficiency.  The  ultimate  result, 
however,  was  similarly  awesome:  warships,  including  the  USS  Constitution,  that  were  the 
most  capable  the  world  had  ever  seen. 

We  can  and  will  build  cutting-edge  equipment  in  the  future,  and,  in  contrast  to  the 
recent  past,  the  current  environment  is  starting  to  encourage  such  development  again.  Even 
if  we  are  always  smart,  such  programs  are  difficult  to  predict:  Some  will  cost  more  than 
expected,  some  will  fail,  and  some  will  be  tremendous  successes.  These  programs  are 
difficult  to  fit  into  40-year  models. 

Conclusion 

Affordability  Analysis  is  a  useful  but  limited  tool  for  the  OSD  to  try  to  make  sure  the 
Services  are  planning  their  acquisitions  far  into  the  future.  Constraints  are  a  part  of  that 
process,  and  allow  the  USD  a  rough  monitor  of  the  affordability  of  each  Service’s  programs 
when  they  are  not  undergoing  DABs.  The  direct  effects  are  likely  positive  but  have  been 
modest. 

The  unexpected  success  is  that  this  initiative  has  brought  the  Service  programmers 
into  the  DABs.  Several  times  in  the  life  of  each  program,  the  program  manager  and  his 
“eight”  sit  in  the  same  room  and  look  at  the  same  long-term  spending  plan.  We  believe  that 
this  is  unprecedented  and  a  significant  benefit  for  the  Department  of  Defense. 
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Abstract 

Institute  for  Defense  Analyses  Paper  P-5126  found  that  additional  acquisition  reforms  after 
those  introduced  in  mid-1969  by  then  Deputy  Secretary  of  Defense  David  Packard  did  not 
significantly  reduce  cost  growth  on  Major  Defense  Acquisition  Programs  (MDAPs).  That 
conclusion — while  interesting — is  incomplete,  as  it  leaves  open  the  possibility  that  the 
Packard  reforms  reduced  cost  growth  compared  to  the  record  of  the  1960s,  which  is  the 
issue  examined  in  this  paper.  The  paper  finds  that  average  cost  growth  of  MDAPs  that 
entered  Engineering  and  Manufacturing  Development  during  fiscal  year  (FY)  1970-FY  1980 
was  significantly  lower  than  the  average  of  those  that  entered  during  FY  1 964-FY  1 969.  It 
also  probably  was  significantly  lower  than  the  average  during  FY  1994-FY  2000  when  Office 
of  the  Secretary  of  Defense  (OSD)-level  oversight  of  MDAPs  was  less  stringent.  These  stand 
as  instances  of  a  significant  association  between  changes  in  OSD-level  oversight  and  cost 
growth.  The  paper  also  provides  evidence  that  average  cost  growth  in  FY  1 964-FY  1969  and 
FY  1994-FY  2000  was  particularly  high  largely  because  the  proportion  of  MDAPs  that 
experienced  extremely  high  cost  growth  was  significantly  larger  than  it  was  in  other  periods. 

Introduction 

McNicol  and  Wu  (2014;  hereafter  referred  to  as  P-5126)  reported  two  significant 
findings.  First,  Major  Defense  Acquisition  Programs  (MDAPs)  that  entered  Engineering  and 
Manufacturing  Development  (EMD)  during  “bust”  funding  climates  on  average  had  much 
higher  cost  growth  than  those  that  entered  EMD  during  “boom”  climates.  Second,  the  paper 
found  that  additional  reforms  after  those  introduced  in  mid-1969  by  then  Deputy  Secretary  of 
Defense  David  Packard  had  not  significantly  reduced  cost  growth. 

As  P-5126  noted,  the  latter  conclusion  leaves  open  the  possibility  that  the  Packard 
reforms  reduced  cost  growth  compared  to  the  record  of  the  1960s.  If  in  fact  they  did,  the 
conclusion  of  that  paper  would  have  to  be  amended  to  read:  The  introduction  in  1969  of 
effective  Office  of  the  Secretary  of  Defense  (OSD)-level  oversight  of  major  acquisition 
programs  reduced  cost  growth,  but  the  additional  reforms  of  the  1970s,  1980s,  and  early 
1990s  did  not  result  in  further  reductions.  Along  the  same  line,  it  is  of  interest  to  revisit  the 
mixed  evidence  P-5126  found  on  the  effect  on  cost  growth  of  less  active  OSD-level 
oversight  of  1994-2000.  The  crucial  question  is  whether  there  is  statistical  evidence  that 
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cost  growth  decreased  when  OSD-level  controls  were  imposed  and  also  increased  when 
those  controls  were  relaxed. 

This  is  not  simply  an  historical  question,  because  the  main  features  of  today’s  OSD 
level  acquisition  oversight  process  remain  those  of  the  process  installed  by  Packard  in  mid- 
1969.  Moreover,  the  issue  is  salient  now  because  of  its  implications  for  ongoing  discussions 
of  reform  of  the  DoD  weapon  system  acquisition  process. 

The  database  available  for  P-5126  did  not  contain  cost  growth  estimates  for  any 
MDAPs  that  entered  EMD  during  the  1960s,  so  that  paper  could  not  compare  cost  growth 
pre-  and  post-Packard.  This  paper  uses  cost  growth  data  for  programs  that  entered  EMD  in 
the  1960s  from  two  previous  studies  (Jarvaise,  Drezner,  &  Norton,  1996;  Tyson,  Om, 
Gogerty,  &  Nelson,  1992).  It  also  uses  a  different  cost  growth  metric  and  employs  additional 
statistical  tests. 

The  next  section  briefly  describes  the  OSD-level  acquisition  oversight  introduced  by 
Robert  McNamara  in  the  mid-1960s  and  the  changes  made  to  it  in  1969  by  Packard.  It  is 
necessary  to  do  this  because  the  McNamara  reforms  are  no  longer  part  of  the  collective 
memory  of  the  DoD  acquisition  community.  Subsequent  sections  then  turn  to  the  statistical 
analysis  and  the  conclusions  it  suggests.  These  sections  assume  that  the  reader  has  a 
working  familiarity  with  acquisition  process  and  policies.  Those  who  do  not  may  wish  to 
consult  Fox  (201 1 ).  Readers  who  want  a  more  detailed  understanding  of  the  data  used  and 
the  way  they  were  binned  should  consult  Appendixes  A  and  B  of  McNicol,  Tate,  Burns,  and 
Wu  (2016). 

Origins  of  the  OSD-Level  Acquisition  Oversight  Process 

From  the  creation  of  the  National  Security  Establishment  in  1947  through  1960,  the 
OSD  had  no  institutionalized  process  for  the  oversight  of  major  weapon  system 
acquisitions.1  The  origins  of  the  OSD-level  process  for  overseeing  major  weapon  system 
acquisitions  lie  in  initiatives  taken  by  McNamara,  of  which  the  following  are  especially 
relevant  for  current  purposes: 

•  Promulgation  of  policy  on  contract  types 

•  Establishment  of  milestone  decision  points  and  the  Development  Concept 
Paper  (DCP) 

•  Active  oversight  of  ongoing  MDAPs2 


1  The  Secretary  of  Defense  could,  and  on  occasion  did,  act  to  cancel  or  initiate  major  acquisitions. 
Major  acquisition  programs  were  also  subject  to  review  during  the  budget  cycle  by  the  Office  of  the 
Assistant  Secretary  of  Defense  (Comptroller)  and  the  Office  of  Management  and  Budget.  Additionally, 
a  major  building  block  of  McNamara’s  process  began  operating  in  1959.  See  O’Neil  and  Porter  (201 1 , 
P-25). 

These  categories  are  abstracted  from  Fox  (201 1 ,  p.  35-45).  Fox  also  notes  that  McNamara  moved 
to  consolidate  acquisition  functions  in  defense  agencies — e.g.,  the  agency  that  became  the  Defense 
Logistics  Agency — and  promoted  the  use  by  program  managers  of  particular  management  tools  such 
as  PERT  and  earned  value.  In  addition,  there  are  several  cases — most  notably  the  F-1 1 1 — in  which 
McNamara  played  a  very  active  role  in  the  oversight  of  the  program.  These  cases  almost  certainly  are 
exceptions,  but  the  literature  survey  done  for  this  paper  uncovered  little  about  how  the  process 
worked  in  the  more  typical  cases.  Adding  to  the  confusion,  the  sources  consulted  suggest  that  during 
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These  initiatives  were  an  embryonic  OSD-level  acquisition  oversight  process. 

McNamara  directed  the  use  of  Total  Package  Procurement  (TPP)  when  it  was 
judged  to  be  practicable  and,  when  not,  Fixed  Price  Incentive  Fee  (FPIF)  or  Cost  Plus 
Incentive  Fee  (CPIF)  contracts.* * 3  By  1966,  McNamara  had  concluded  that  TPP  contracts 
were  in  fact  not  a  practicable  way  to  acquire  major  weapon  systems,  although  acquisition 
policy  apparently  still  had  a  tilt  towards  fixed  price  contracts,  even  for  development.  Packard 
picked  up  on  this  topic  where  McNamara  left  off.  He  ruled  out  the  use  of  TPP  and 
discouraged  the  use  of  FPIF  for  development  contracts  in  favor  of  CPIF.  (Cost  Plus  Award 
Fee  may  not  have  been  included  in  the  contracting  play  book  yet.)  As  a  general  matter, 
Packard’s  policy  was  to  match  contract  terms  to  the  riskiness  of  the  acquisition. 

Packard’s  establishment  of  the  Defense  Systems  Acquisition  Review  Council 
(DSARC)  often  is  seen  as  the  hallmark  of  his  1969  reforms.  The  notion  of  milestone  reviews, 
however,  entered  the  OSD-level  acquisition  process  in  1964  with  issuance  of  DoD  Directive 
(DoDD)  3200.9,  Initiation  of  Engineering  and  Operational  Systems  Development.4  This 
original  version  of  the  directive  set  one  point  at  which  OSD — in  principle,  the  Secretary  of 
Defense — approval  was  required  for  an  acquisition  program  to  proceed.  In  1965,  a  second 
decision  point  was  added,  and  the  Director,  Defense  Research  and  Engineering  (DDR&E), 
instituted  the  precursor  of  the  DCP,  which,  starting  in  1968,  was  required  to  initiate  any 
major  development  project.  DDR&E  coordinated  initial  DCPs  with  concerned  OSD  offices 
(and  probably  the  Joint  Staff  and  other  Services;  O’Neil  &  Porter,  201 1 )  and  acted  as  what 
now  would  be  called  the  Milestone  Decision  Authority  (MDA)  for  the  initial  DCP  (Borklund, 
1969).  Once  approved  by  DDR&E,  the  proposed  new  start  went  to  the  Secretary  of 
Defense,  although  the  sources  consulted  do  not  indicate  whether  it  went  as  a  separate 
action  or  as  part  of  the  Service’s  budget  submission.  It  is  also  not  clear  which  OSD  official 
was  the  MDA  for  the  second  milestone. 

Viewed  against  this  background,  the  establishment  of  the  DSARC  was  an 
evolutionary  step.  The  Development  Concept  Paper  was  renamed  the  Decision 
Coordinating  Paper  (retaining  the  acronym)  to  reflect  the  broader  scope  of  the  new 
milestone  definitions.  The  MDA  at  Milestone  (MS)  I  and  MS  II  was  DDR&E;  the  Assistant 
Secretary  of  Defense  for  Installations  and  Logistics  was  the  MDA  for  MS  III.  Decisions  at  the 
DSARC  level  were  advisory  to  the  Secretary  and  Deputy  Secretary  of  Defense  but,  apart 
from  exceptional  cases,  they  probably  reached  that  level  by  way  of  the  Service’s  proposed 
budgets  (and  the  Comptroller  was  the  backstop  enforcer  of  the  requirement  for  milestone 
approval  before  a  program  could  advance  to  the  next  stage). 

The  OSD  had  a  much  larger  role  in  oversight  of  major  acquisition  programs  under 
the  DSARC  process  than  it  did  pre-1961 .  The  picture  in  contrast  to  the  McNamara  years  is 


the  McNamara  years  a  major  acquisition  program  might  arise  in  either  the  Planning,  Programming, 

and  Budgeting  System  (PPBS)  or  in  the  acquisition  process. 

3  Fox  (2011,  p.  38),  following  Adams,  Murphy,  and  Rosenau  (1983,  pp.  19-20).  A  TPP  contract  is  one 
that  covers  EMD,  at  least  a  significant  portion  of  procurement,  and  at  least  part  of  the  support  of  the 
system  (e.g.,  depot  maintenance). 

4  The  first  version  of  DoDD  3200.9  was  issued  in  1964.  A  revision  that  made  provision  for  the 
Contract  Definition  Phase  was  issued  July  1,  1965.  See  Glennan  (1965,  p.  12).  O’Neil  and  Porter 
(2011,  pp.  25-47)  sketch  how  the  process  evolved  and  worked  during  the  1960s. 
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less  clear-cut.  On  one  hand,  under  the  new  acquisition  directives,  the  Secretary  of  Defense, 
while  retaining  full  legal  authority  over  acquisition  programs,  would  act  through  the 
established  acquisition  process  except  in  extraordinary  circumstances,  which  in  comparison 
to  cases  such  as  the  F-1 1 1  implied  less  OSD-level  control  over  acquisitions.  On  the  other 
hand,  the  DSARC  had  a  greater  substantive  scope  for  the  more  typical  program  and  was 
more  tightly  organized.  For  the  large  majority  of  major  acquisition  programs,  then,  the  new 
DSARC  process  probably  was  more  effective.5 

The  most  consequential  of  Packard’s  1969  reforms  involved  the  substance  of  the 
milestones.6  The  1965  version  of  the  DoDD  3200.9  process  had  three  phases.  The  first  of 
these  “was  called  concept  formulation.  During  concept  formulation  OSD  and  the  Service(s) 
involved  assured  themselves  that  they  were  buying  the  right  system  to  meet  real  needs  and 
that  the  technology  was  fully  ready”  (O’Neil  &  Porter,  201 1 ,  p.  30).  Concept  formulation 
typically  was  initiated  by  a  Service  but  involved  DDR&E  and  the  Office  of  the  Assistant 
Secretary  of  Defense  for  Systems  Analysis  (OASD[SAj),  and  included  what  would  now  be 
called  an  Analysis  of  Alternatives  led  by  OASD(SA).  It  also  apparently  included  what  would 
later  be  called  a  Mission  Element  Need  Statement  as  well  as  the  main  parts  of  an 
Acquisition  Strategy  and  plans  for  oversight  of  the  program  as  it  proceeded. 

Approval  to  proceed  from  the  Concept  Formulation  phase  authorized  the  Service 
sponsoring  the  program  to  fund  at  least  one  company  to  prepare  a  definitized  contract 
proposal.  The  OSD  (milestone)  review  of  these  proposals  was  the  basis  for  award  of  a 
contract,  usually  to  a  single  source,  for  development  and  procurement  of  the  system.  That  is 
to  say,  the  second  of  DoDD  3200. 9’s  milestones  combined  what  now  would  be  called  MS  B 
and  MS  C  authority. 

Packard’s  reforms  separated  the  decision  to  allow  the  program  to  enter  EMD  from 
the  decision  to  enter  the  production  phase  (now  MS  C)  and  required  OSD-level  approval  of 
each  decision.  Packard  also  established  a  new  Validation  Phase,  which  has  at  various  times 
since  been  called  Demonstration  and  Validation,  Program  Development  and  Risk  Reduction 
and,  currently,  Technology  Maturation  and  Risk  Reduction.  MS  I  (now  MS  A)  authorized 
entry  into  this  phase.  DoDD  3200. 9’s  Contract  Definition  phase  was  collapsed  into  the  new 
and  broader  Validation  phase.  These  changes  were  more  revolutionary  than  evolutionary.7 

The  provisional  judgment  offered  here  is  that  Packard’s  acquisition  reforms  provide  a 
plausible  reason  for  expecting  program  outcomes — measured  by  cost  growth,  schedule 
slips,  and  performance  shortfalls — to  be  better  than  what  was  achieved  during  the 


5  Murdock  (1974,  pp.  155-179),  disagrees  with  this  judgment.  Murdock  is  primarily  concerned  with 
Systems  Analysis  and  resource  allocation,  but  also  comments  specifically  on  the  acquisition  process. 
In  particular,  he  notes  that  the  new  Decision  Coordinating  Paper  did  not  provide  “any  mechanism  for 
ongoing  managerial  control.”  This  is  accurate  in  that  the  Packard  reforms  placed  management  of  the 
programs  in  the  hands  of  the  Services.  It  is  incomplete  in  that  the  Services  were  responsible  for 
staying  within  what  would  later  be  called  the  Acquisition  Program  Baseline,  and  the  MDA  was 
enjoined  to  act  in  cases  in  which  they  did  not. 

6  Fox  (201 1 ,  p.  57),  provides  a  useful  schematic  comparison  of  the  DoDD  3200.9  milestones  and 
those  of  Packard’s  DoDD  5000.1/DoDI  5000.2. 

7  DoDI  5000.2,  issued  October  23,  2000,  formally  established  MSs  A,  B,  and  C  (replacing  MSs  I,  II, 
and  III)  as  the  main  decision  points  for  an  MDAP.  The  definitions  are  such  that  MS  B  is  placed 
several  months  earlier  in  the  process  than  MS  II. 
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McNamara-Clifford  years.  This  judgment  does  not  imply  that  the  DoD  was  doing  a  better  job 
of  deciding  what  to  buy,  but  only  that,  as  a  result  of  the  Packard  reforms,  the  OSD  became 
more  effective  in  oversight  of  acquisition  programs  from  MS  II  through  the  completion  of 
procurement. 

Statistical  Analysis  of  Average  Cost  Growth 

The  statistical  analysis  presented  here  rests  on  definitions  of  periods  delimited  by 
major  changes  in  acquisition  policy  and  process.  Two  of  these  already  encountered  are 
labeled  “McNamara-Clifford”  and  “DSARC.”  Four  additional  acquisition  periods  are 
introduced  below.  Another  part  of  the  scaffolding  of  the  analysis  is  funding  climate.  Two 
climates  are  distinguished — “bust”  and  “boom.”  Three  of  the  acquisition  periods  include  both 
bust  and  boom  phases  and  three  were  entirely  in  a  single  funding  climate.  Finally,  the 
analysis  rests  on  a  set  of  conventions  concerning  which  MDAPs  are  included  in  the 
database  and  the  way  in  which  cost  growth  is  measured.  See  Appendix  A  of  McNicol  et  al. 
(2016)  for  an  explanation  of  the  basis  of  the  boundaries  separating  the  successive 
acquisition  periods  and  the  funding  climates.  Appendix  B  of  McNicol  et  al.  states  the 
conventions  used  in  assembling  the  database  and  identifies  the  sources  of  the  data  used. 

This  section  considers  whether  there  are  statistically  significant  differences  in  cost 
growth  across  the  successive  acquisition  regimes  in  bust  climates.  The  measure  of  cost 
growth  used  is  Average  Procurement  Unit  Cost  (APUC).  “APUC  growth”  means  growth  in 
APUC  in  program  base  year  dollars  normalized  to  the  baseline  quantity  approved  at  MS  B. 
Attention  in  this  section  and  most  of  the  one  that  follows  is  limited  to  MDAPs  that  entered 
EMD  during  bust  periods  because  the  interesting  findings  arise  from  the  analysis  of  those 
periods.  Results  for  boom  periods  are  briefly  mentioned  at  the  end  of  the  following  section. 

Table  1  reports  average  APUC  growth  experienced  by  MDAPs  that  passed  MS  ll/B 
during  each  of  the  six  acquisition  regimes  in  a  bust  climate.  It  is  important  to  bear  in  mind 
that  APUC  growth  is  computed  by  comparing  the  MS  ll/B  baseline  value  for  APUC — which 
can  be  thought  of  as  a  goal  or  a  prediction — to  the  actual  APUC,  normalized  to  the  MS  ll/B 
quantity8  (or,  for  ongoing  programs,  to  the  projected  APUC  in  the  December  2012  Selected 
Acquisition  Reports  [SARs],  which  were  the  most  recent  available  when  this  project  began).9 
The  APUC  growth  figures  shown  are  the  quantity  normalized  average  for  the  MDAPs  in  that 
acquisition  regime,  binned  by  the  year  the  MDAP  passed  MS  ll/B.  This  is  done  on  the 
hypothesis  that  the  acquisition  policies  and  processes  in  place  when  an  MDAP  passes  MS 
ll/B,  particularly  the  rigor  of  the  MS  ll/B  review,  have  an  effect  on  the  amount  of  cost  growth 
it  experiences  in  the  future. 


8  About  three-quarters  of  the  MDAPs  that  passed  MS  ll/B  in  the  period  FY  1988-FY  2007  acquired  at 
least  90%  of  their  MS  ll/B  baseline  quantity.  The  median  program  acquired  100%  and  the  average 
program  acquired  111%.  See  McNicol  et  al.  (2015,  p.  7-8). 

9  We  follow  the  convention  of  not  including  in  the  database  any  MDAP  that  was  not  at  least  five  years 
beyond  EMD  (so  that  cost  growth  would  have  time  to  appear).  The  most  recent  SARs  available  when 
P  5126  was  written  were  those  for  December  2012.  Consequently,  MDAPs  that  passed  MS  B  during 
FY  2007  were  the  most  recent  included  in  the  database. 
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Table  1.  Average  APUC  Growth  by  Acquisition  Regime  for  MDAPs  That  Entered 

EMD  During  a  Bust  Funding  Climate 


Acquisition  Regime 

Period  (FY) 

Average  APUC 
Growth* 

McNamara-Clifford 

1964-1969 

85%  (20) 

DSARC 

1970-1980 

39%  (53) 

Post  Carlucci  DSARC 

1987-1989 

44%  (12) 

Defense  Acquisition  Board  (DAB) 

1990-1993 

32%  (11) 

Acquisition  Reform  (AR) 

1994-2000 

78%  (27) 

DAIB  post  AR 

2001-2002 

113%  (6) 

Note.  Numbers  in  parentheses  are  the  number  of  observations  in  the  cell. 

*  Normalized  for  changes  in  quantity. 

A  plausible  reading  of  the  averages  in  Table  1  is  as  follows:  Packard’s  radically  new 
acquisition  phases  and  his  more  highly  structured  process  were  successful  in  reducing 
APUC  growth,  which  fell  to  less  than  half  the  average  level  it  had  during  the  1960s.  Perhaps 
encouraged  by  Packard’s  success  and  public  distaste  for  cost  growth,  acquisition  reform 
efforts  persisted,  but  had  no  appreciable  further  effect  on  average  cost  growth  prior  to  the 
AR  years.  Reduction  of  OSD  oversight  during  the  AR  era  coincided  with  the  return  of 
average  APUC  growth  to  nearly  its  1960s  level.  In  sum,  the  Packard  reforms  of  late  FY  1969 
appear  to  have  reduced  APUC  growth;  they  were  not  significantly  improved  upon  in  this 
respect  through  the  bust  years  that  followed;  and  the  AR  years  were  associated  with  higher 
APUC  growth,  which  may  be  related  to  a  reduction  of  OSD-level  oversight. 

The  question  for  the  statistical  analysis  in  an  exploratory  context  is:  Can  cause 
reasonably  be  ascribed  to  the  period-to-period  changes  in  APUC  growth,  or  are  those 
changes  more  likely  simply  random  fluctuations  in  the  data? 

It  is  useful  to  break  this  question  into  three  parts.  First,  is  the  difference  between  the 
average  APUC  growth  post  Packard  reforms  (39%)  and  the  average  for  FY  1964-FY  1969 
(85%)  statistically  significant?  The  tests  used  found  this  difference  to  be  statistically 
significant  at  the  9%  level.10  It  is  worth  noting  these  reductions  probably  cannot  be  attributed 
only  to  the  policies  on  contract  type  that  Packard  instituted.  Four  of  the  20  programs  in  the 
data  set  for  FY  1964-FY  1969  used  TPP,  and  one  used  a  Firm  Fixed  Price  (FFP) 
development  contract.  The  average  APUC  growth  for  these  five  contracts  was  131%;  the 
average  cost  growth  for  the  remaining  FY  1961-FY  1969  programs  was  70%.*  11  TPP  and 
FFP  contracts  were  less  commonly  used  during  FY  1970-FY  1980,  but  three  of  the  MDAPs 


10  The  Mann-Whitney  U  test  rejected  the  null  hypothesis  (P  =  0.093)  that  the  samples  for  the  DSARC 
period  and  the  McNamara-Clifford  period  were  drawn  from  the  same  population,  (m  =  53,  n2  =  20,  U 
=  394).  A  two-tail  t-test  assuming  unequal  sample  variances  found  the  difference  in  the  means  to  be 
significant  (p  =  0.074).  The  Kolmogorov-Smirnov  (K-S)  test  showed  that  APUC  growth  estimates  for 
the  McNamara-Clifford  period  probably  are  not  normally  distributed.  The  result  of  the  t-test,  even  with 
the  correction  for  unequal  variances,  is  therefore  somewhat  suspect. 

11  For  further  discussion  of  TPP  and  FFP  development  contracts,  see  Tyson  et  al.  (1992,  Chapter  X); 
McNicol  (2004,  pp.  53,  57-59);  and  O’Neil  and  Porter  (2011,  p.  29-31). 
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that  passed  MS  ll/B  during  this  period  used  a  TPP  contract  and  one  used  an  FFP 
development  contract. 

Second,  are  the  differences  in  average  APUC  growth  for  the  three  periods  between 
McNamara-Clifford  and  AR  statistically  significant?  The  tests  used  did  not  reveal  any 
statistically  significant  differences  between  the  averages  of  APUC  growth  in  these  three 
periods.12  This  implies  that  the  lower  average  APUC  growth  (32%)  of  MDAPs  that  passed 
MS  II  during  the  DAB  years  (FY  1990-FY  1993),  for  example,  cannot  be  attributed 
confidently  to  the  full  implementation  of  the  DAB  in  1990,  because  a  change  of  this  size  has 
a  considerable  probability  of  occurring  by  chance. 

Third,  and  finally,  were  the  AR  years  associated  with  significantly  higher  average 
APUC  growth?  The  results  in  this  case  were  mixed.  One  test  indicated  that  average  APUC 
growth  over  the  AR  years  was  significantly  higher  than  it  was  in  FY  1990-FY  1993.  That 
result,  however,  was  not  confirmed  by  another  test.13  This  is  similar  to  the  result  found  in  P- 
5126  and  it  occurs  for  the  same  reason — the  variability  of  APUC  growth  in  the  AR  period 
was  too  large  for  the  differences  in  the  means  to  be  statistically  significant. 

The  Bayesian  analysis  presented  in  Appendix  C  of  McNicol  et  al.  (2016)  provides  a 
stronger  result  for  the  AR  years.  It  finds  clear  evidence  that  both  the  McNamara-Clifford 
period  (FY  1964-FY  1969)  and  the  AR  years  (FY  1994-FY  2000)  had  a  much  higher 
probability  of  high  cost  growth  than  did  the  bust  climate  portion  of  any  of  the  three 
intervening  periods  (DSARC,  Post-Carlucci  DSARC,  and  DAB). 

Returning  to  the  interpretation  of  Table  1  offered  above,  the  statistical  analysis  of 
average  APUC  growth  supports  two  of  the  three  points  offered  above — the  Packard  reforms 
did  reduce  APUC  growth  and  the  further  reforms  introduced  post-Packard  and  pre-AR  did 
not  yield  significant  further  reductions  in  APUC  growth.  The  results  on  the  third  point  are  not 
clear-cut.  The  statistical  tests  reported  above  do  not  support  attributing  the  high  mean 
APUC  growth  during  FY  1994-FY  2000  to  acquisition  reform,  but  the  Bayesian  analysis 
does  support  such  an  interpretation. 

Statistical  Analysis  of  the  Proportion  of  Extremely  High  APUC  Growth  Programs 

The  preceding  section  looked  for  effects  of  acquisition  policy  and  process  in 
differences  between  successive  periods  in  the  average  APUC  growth  of  MDAPs  that  passed 
MS  ll/B  during  them.  Although  reasonable,  framing  the  analysis  in  this  way  glosses  over  the 
possibility — explored  in  this  section — that  acquisition  policy  and  process  mainly  work  by 
influencing  the  proportion  of  MDAPs  that  experience  extremely  high  cost  growth. 

Some  relevant  data  are  provided  in  Table  2.  The  average  APUC  growth  figures  are 
the  same  as  those  presented  in  Table  1.  In  addition,  Table  2  reports  the  number  of  MDAPs 


12  We  compared  the  three  periods  using  Analysis  of  Variance  (ANOVA).  ANOVA  failed  to  reject  the 
null  hypothesis  that  the  observations  in  the  three  periods  were  drawn  from  identical  normal 
populations.  The  K-S  test  found  it  highly  likely  that  the  samples  were  consistent  with  ANOVA’s 
assumptions. 

13  A  two-tail  t-test  assuming  unequal  variances  found  the  difference  to  be  significant.  (P  =  0.084.)  The 
K-S  test  rejected  the  null  hypothesis  that  the  observations  for  FY  1994-FY  2000  were  normally 
distributed.  A  Mann-Whitney  U  test  did  not  find  a  significant  difference  between  the  average  APUC 
growth  of  the  AR  years  and  that  for  the  period  FY  1990-FY  1993. 
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in  the  cohort  that  experienced  at  least  three  different  levels  of  APUC  growth — 50%,  100%, 
and  one  standard  deviation  (S)  above  the  sample  mean  (X).  The  sample  mean  is  57.4%  and 
the  standard  deviation  is  85.4%,  so  one  standard  deviation  beyond  the  mean  is  143%.  (X 
and  S  are  computed  for  the  bust  periods  only.)  In  what  follows,  MDAPs  in  the  last  of  the 
categories  will  be  called  “extremely  high  cost  growth”  programs.  These  are  arbitrary  breaks 
adopted  because  they  proved  to  be  useful.  Note  that  the  figures  for  the  number  of  systems 
in  the  right  tail  are  not  additive.  For  example,  of  the  20  MDAPs  that  entered  EMD  during  the 
period  FY  1964-FY  1969,  10  had  APUC  growth  of  at  least  50%.  Of  these  10,  six  had  APUC 
growth  of  more  than  100%,  and  of  the  six,  four  had  APUC  growth  of  more  than  143%. 

The  striking  feature  of  the  data  in  Table  2  is  the  paucity  of  extremely  high  cost  growth 
programs  after  the  introduction  of  the  Packard  reforms  in  1969  and  before  AR.  A  total  of  76 
programs  in  our  sample  passed  MS  II  during  the  18  years  of  the  DSARC,  Post-Carlucci 
DSARC,  and  DAB  periods  in  bust  funding  climates.  Only  one  of  these  has  an  estimated 
quantity  normalized  APUC  growth  from  the  MS  II  baseline  of  at  least  143%. 14  The  other  side 
of  this  coin  is  the  greater  frequency  of  extremely  high  cost  growth  systems  in  the 
McNamara-Clifford  years  and  during  the  AR  period.  Four  out  of  20  programs  of  the 
McNamara-Clifford  years  showed  extremely  high  cost  growth,  as  did  seven  out  of  27 
MDAPs  that  passed  MS  II  during  the  AR  years. 


Table  2.  Average  APUC  Growth  by  Acquisition  Regime  and  the  Number  of  High 
Cost  Growth  MDAPs  in  Each  Cohort,  Bust  Funding  Climates 


Acquisition  Regime 

Period 

(FY) 

Average 

APUC 

Growth* 

>  50% 

>  100% 

>  X  -  S 

McNamara-Clifford 

1964-1969 

85%  (20) 

10 

6 

4 

DSARC 

1970-1980 

39%  (53) 

19 

7 

0 

Post-Carlucci  DSARC 

1987-1989 

44%  (12) 

4 

3 

1 

DAB 

1990-1993 

32%  (11) 

5 

1 

0 

Acquisition  Reform 

1994-2000 

78%  (27) 

11 

7 

7 

DAB  post-AR 

2001-2002 

113%  (6) 

2 

1 

1 

Note.  Numbers  in  parentheses  are  the  number  of  observations  in  the  cell. 

*  Normalized  for  changes  in  quantity. 

Statistical  analysis  gives  substantially  the  conclusions  suggested  by  inspection  of  the 
data  in  Table  2: 

•  The  frequency  of  extremely  high  cost  growth  programs  was  significantly 
higher  in  the  McNamara-Clifford  years  than  in  the  DSARC  period. 


14  This  is  the  FGM-148A  Javelin.  Roland  also  had  a  very  high  APUC  growth  (308%)  but  was  placed 
on  the  cancelled  list.  Roland  was  developed  during  the  mid-1960s  by  a  French-German  consortium. 
In  1975,  the  U.S.  Army  decided  to  develop  and  procure  a  U.S.  version.  The  planned  procurement 
was  severely  reduced,  but  enough  was  acquired  to  equip  one  Army  National  Guard  battalion.  This 
does  not  fully  meet  the  definition  of  a  cancellation  but  was  judged  to  be  closer  to  a  cancellation  than 
to  a  truncation  of  the  program. 
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•  The  frequency  of  extremely  high  cost  growth  programs  also  was  significantly 
higher  during  the  AR  years  than  during  the  DSARC  period.15 

In  contrast  to  the  results  of  the  preceding  section,  both  the  McNamara-Clifford  period 
and  the  AR  period,  then,  stand  out  as  having  a  significantly  larger  proportion  of  extremely 
high  cost  growth  programs. 

Table  3  lists  the  extremely  high  cost  growth  systems.  Thirteen  of  the  14  passed  MS 
ll/B  during  bust  climates.  Helicopters  (2),  satellite  programs  (3),  and  launch  vehicles  (2)  are 
over-represented  but  do  not  dominate  the  list,  particularly  for  the  1960s. 


Table  3.  Extremely  High  Cost  Growth  Systems 


System  Name 

MS  ll/B  FY  APUC 

Bust  Climates 

AGM-69  Short  Range  Attack  Missile  (SRAM) 

1967 

4.56 

MIM-23  Hawk  (Improved  Hawk) 

1965 

2.07 

Versatile  Avionics  Shop  Test  (VAST) 

1968 

1.83 

M47  Dragon  Guided  Missile 

1966 

1.72 

FGM-143A  Javelin  Advanced  Anti-Tank  Weapon  System 

1939 

1.59 

Space  Based  IR  Sensor  (SBIRS)  High 

1997 

3.90 

Evolved  Expendable  Launch  Vehicle  (EELV) 

1998 

3.42 

Global  Broadcast  Service  (GBS) 

1993 

2.60 

Guided  Multiple  Launch  Rocket  System  (GMLRS) 

1993 

2  15 

H-1  Upgrades 

1998 

1.97 

CH-47IF  (Improved  Cargo  Helicopter) 

1998 

1.81 

Patriot  Advanced  Capability-3  (PAC-3) 

1994 

1.49 

Advanced  Extremely  High  Frequency  (AEHF)  Satellite 

2001 

4.73 

Boom  Climates 

Titan  IV  Expendable  Launch  Vehicle  {ELV) 

1985 

1.49 

We  also  explored  whether  the  proportions  of  systems  with  cost  growth  of  at  least 
50%  or  100%  might  show  the  same  pattern  across  acquisition  periods  as  the  extremely  high 
cost  growth  systems.  Analyses  parallel  with  those  just  described,  with  observations  of  at 
least  50%  APUC  growth  and  100%  APUC  growth  showing  no  significant  differences  across 
the  acquisition  periods. 

Appendix  D  of  McNicol  et  al.  (2016)  presents  results  obtained  from  a  technique 
(quantile  regression)  that  compares  the  APUC  growth  distributions  across  acquisition 
regimes  at  several  points.  The  comparison  reported  used  deciles.  The  results  were 


15  These  statements  are  based  on  results  for  Fisher’s  Exact  Tests:  (1 )  p  =  0.004  in  the  comparison  of 
McNamara-Clifford  to  the  DSARC  years,  and  (2)  p  <  0.001  for  the  comparison  of  FY  1994-FY  2000 
with  the  DSARC  years. 
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consistent  with  those  stated  above  in  two  respects:  (1)  There  were  no  significant  differences 
across  the  six  acquisition  periods  in  the  central  portions  of  the  distribution  (4th  through  the 
7th  deciles),  and  (2)  the  McNamara-Clifford  and  AR  periods  had  significantly  fatter  right  tails. 
It  also  is  interesting  to  note  that  there  is  some  evidence  that  the  left  tails  of  these  two 
periods  were  somewhat  fatter  than  those  of  other  periods;  that  is,  McNamara-Clifford  had 
higher  highs  and  perhaps  higher  lows. 

Finally,  we  considered  the  pattern  in  average  APUC  growth  across  the  six  acquisition 
periods  if  the  13  extremely  high  cost  growth  programs  are  removed.  The  means  of  the 
truncated  distributions  are  presented  in  Table  4.  Pair-wise  tests  found  the  average  APUC 
growth  for  the  AR  years  (without  the  extremely  high  cost  growth  systems)  to  be  significantly 
lower  than  the  averages  for  the  McNamara-Clifford  and  DSARC  periods.  None  of  the  other 
differences  was  statistically  significant  and  a  test  of  the  table  as  a  whole  did  not  reveal 
significant  differences.16  It  appears,  then,  that  the  significant  differences  in  average  APUC 
growth  reported  in  the  previous  section  (Statistical  Analysis  of  Average  Cost  Growth)  stem 
from  the  significantly  higher  proportion  of  extremely  high  cost  growth  systems  during  the 
McNamara-Clifford  and  AR  periods. 

Table  4.  Average  APUC  Growth  by  Acquisition  Regime  for  Bust  Funding 
Climates,  Excluding  Extremely  High  Cost  Growth  Programs 


Acquisition  R&gime 

Period  (FY) 

Average  APUC 
Growth* 

McNamara-Clifford 

1964-1969 

0.43(16} 

DSARC 

1970 -I960 

0.39  (53} 

Post  Carlucci  DSARC 

1937-1989 

0.34(11} 

DAB 

1990-1993 

0.32(11} 

Acquisition  Reform 

1994-2000 

0.18(20} 

DAB  post-AR 

2001-2002 

0.40  (5) 

Note.  Numbers  in  parentheses  are  the  number  of  observations  in  the  cell. 

*  Normalized  for  changes  in  quantity. 

Appendix  E  of  McNicol  et  al.  (2016)  presents  an  analysis  of  the  boom  case  that 
parallels  that  of  this  and  the  preceding  section  for  the  bust  case.  There  was  no  indication  of 
significant  association  between  acquisition  period  and  average  APUC  growth  and  no 
indication  of  statistically  significant  differences  across  the  acquisition  regimes  in  the  boom 
periods  with  the  proportion  of  MDAPs  in  the  right  tail  of  the  distributions. 

Interpretation  of  the  Statistical  Results 

The  conclusions  of  the  preceding  section  add  a  level  of  detail  to  the  interpretation  of 
the  APUC  growth  data  offered  in  the  earlier  section  titled  Statistical  Analysis  of  Average 


16  Two-tail  t-test  of  the  differences  of  the  means  of  two  independent  samples.  ANOVA  for  the  table  as 
a  whole  yielded  P  =  0.45.  K-S  found  four  of  the  distributions  to  be  normal.  The  exceptions  were  those 
for  FY  1964-FY  1969,  which  K-S  found  to  only  marginally  satisfy  the  test  for  normality,  and  FY  2001- 
FY  2002,  which  had  too  few  data  points  to  test. 
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Cost  Growth.  Packard’s  radically  new  acquisition  phases  and  his  more  highly  structured 
process  were  almost  completely  successful  in  preventing  instances  of  extremely  high  cost 
growth  and,  for  this  reason,  significantly  reduced  average  APUC  growth.  The  relaxation  of 
OSD  oversight  of  MDAPs  during  the  AR  era  saw  a  return  of  a  significant  number  of 
extremely  high  cost  growth  systems  and,  for  that  reason,  average  APUC  growth  returned  to 
nearly  its  1960s  level.  In  sum,  the  Packard  reforms  of  late  FY  1969  worked  well  in 
essentially  eliminating  instances  of  extremely  high  cost  growth  and  in  that  way  reduced 
average  APUC  growth;  they  were  not  significantly  improved  upon  in  this  respect  through  the 
early  2000s;  and  the  relaxation  of  OSD-level  oversight  of  the  AR  years  was  associated  with 
a  significant  number  of  extremely  high  cost  growth  programs  and,  therefore,  of  higher 
average  APUC  growth. 

The  DAB  process  is  a  mechanism  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  can  use  to  bring  MDAPs  into  conformance  with  acquisition  policy 
at  MS  ll/B.  Among  other  things,  programs  should  have  use  the  appropriate  contracting 
mechanism,  should  have  a  sound  test  plan,  should  not  proceed  until  the  technologies  to  be 
employed  are  reasonably  mature,  should  rest  on  realistic  programmatic  assumptions,  and 
should  be  fully  funded  to  a  realistic  cost  estimate.  It  is  not  surprising,  then,  to  find  that 
(except  in  the  AR  years  when  OSD-level  oversight  was  relaxed)  the  DSARC  process  and  its 
successor,  the  DAB  process,  largely  eliminated  instances  of  extreme  cost  growth.  This 
might  be  due  to  direct  OSD-level  modification  of  particular  MDAPs.  Alternatively,  the 
certainty  of  reviews  by  the  DSARC/DAB  might  have  prompted  the  Services  to  avoid  in  the 
programs  they  proposed  the  characteristics  that  cause  high  cost  growth.  The  best  way  to 
gain  a  deeper  insight  into  the  matter  probably  is  to  compare  closely  the  AR  period  with  the 
DSARC  period  and  to  examine  the  extremely  high  cost  growth  programs. 

It  is  surprising  that  the  statistically  significant  differences  are  found  only  for  the 
extremely  high  cost  growth  systems.  The  description  of  the  process  certainly  suggests  that  it 
also  should  have  an  effect  on  programs  with  smaller  but  still  very  substantial  cost  growth. 
This  finding,  however,  does  not  necessarily  imply  that  the  OSD-level  process  has  no  effect. 
Instead,  the  statistical  finding  as  such  is  that  the  fairly  rudimentary  OSD-level  process  of  the 
McNamara-Clifford  years  did  as  well  as  its  more  elaborate  successors  except  on  extremely 
high  cost  growth  systems. 

It  is,  finally,  important  to  note  that  this  paper  has  been  concerned  almost  entirely  with 
cost  growth  of  MDAPs  that  passed  MS  ll/B  in  bust  periods.  A  complete  summary  also  would 
need  to  take  into  account  parallel  analyses  for  the  boom  periods  and  the  comparisons  of 
cost  growth  in  bust  and  boom  periods  for  a  given  acquisition  regime.  That  task,  however,  is 
postponed  to  a  subsequent  study. 
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Abstract 

Senior  national  security  leaders  face  a  diverse  set  of  threats  and  greater  uncertainty  than  in 
the  past.  They  have  called  for  adaptable  or  agile  organizations  and  weapon  systems  to 
address  this  uncertainty.  We  focus  on  what  this  means  for  weapon  system  acquisition  in 
terms  of  design,  threats,  and  processes.  Additionally,  we  show  how  metrics  can  be 
quantitatively  used  to  help  leadership  understand  the  costs  and  benefits  of  adaptable  and 
non-adaptable  weapon  systems. 

Introduction 


The  United  States  is  going  to  maintain  our  military  superiority  with  armed 
forces  that  are  agile,  flexible,  and  ready  for  the  full  range  of  contingencies 
and  threats. 

— President  Obama,  January  5,  2012 

Background 

The  imperative  for  U.S.  forces  to  be  adaptive  to  changing  circumstances  is  driven  by 
uncertainty  regarding  potential  threats  and  operational  environments,  coupled  with  likely 
reductions  in  force  structure  and  modernization  accounts.1  In  many  disciplines,  time  and 
time  again,  it  has  been  demonstrated  that  expectations  regarding  the  future  are  often 
wrong — sometimes  very  wrong,  resulting  in  severe  consequences.  The  Department  of 
Defense  (DoD)  has  not  been  immune  from  this  tendency.  The  modesty  these  failures  should 
engender  is  manifested  in  the  importance  accorded  the  idea  of  adaptability  in  recent  pre¬ 
eminent  strategic  guidance  documents.2  Senior  leaders  are  directing  the  DoD  to  prepare  to 
be  wrong.  This  perspective  raises  several  questions:  What  is  an  appropriate  conceptual 
definition  of  adaptability  for  the  DoD?  How  does  that  definition  apply  to  the  different 
functions  of  the  Department?  And  how  could  you  operationalize  and  measure  it  in  those 
functions?  The  first  two  questions  have  received  some  attention,  the  latter  far  less. 

Not  surprisingly,  the  concept  of  adaptability  has  recently  been  scrutinized  and 
considered  within  a  DoD  context.  An  enterprise-level  definition  used  by  the  Defense  Science 
Board  (DSB;  201 1 )  is  “the  ability  and  willingness  to  anticipate  the  need  for  change,  to 


1  For  example,  now  and  in  the  future,  there  are  no  fewer  than  five  interdependent  domains  for 
warfare:  land,  sea,  air,  space,  and  cyberspace.  It  has  been  rare  in  history  for  a  new  domain  to  be 
added  to  the  short  list  of  environments  for  warfare,  and  yet  two  such  new  domains,  space  and 
cyberspace,  were  added  only  recently  (Gray,  2008-2009). 

2  See,  for  example,  Office  of  the  Secretary  of  Defense  (2012)  and  Dempsey  (2012). 
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prepare  for  that  change,  and  to  implement  changes  in  a  timely  and  effective  manner  in 
response  to  the  surrounding  environment”  (p.  viii).  With  this  definition  in  hand,  the  DSB 
(2011,  p.  30)  reviewed  the  DoD  enterprise  and  offered  several  recommendations,  two  of 
which  motivated  this  paper:  first,  the  call  to  align  processes  to  the  pace  of  today’s 
environment — more  specifically,  to  employ  dynamic  trade  space  analysis;  and  second,  to 
reduce  uncertainty  through  better  awareness.  Regarding  the  second,  however,  the 
approach  taken  here  assumes  that  the  DoD  will  make  little  progress  in  this  regard  and, 
therefore,  should  place  equal  if  not  more  emphasis  on  explicitly  accounting  for  uncertainty  in 
its  capability  development  and  acquisition  processes. 

In  Operation  Iraqi  Freedom  and  Operation  Enduring  Freedom,  U.S.  forces 
encountered  an  agile  enemy  adapting  quickly  in  the  tactical  arena.  In  such  operational 
environments,  survival  requires  a  local  response.  Success,  however,  depends  on  rapid 
response  at  all  DoD  enterprise  levels  (DSB,  201 1 ,  p.  viii).  In  some  instances,  changes  in  the 
way  our  warfighters  engage  the  adversary — modifying  tactics,  techniques,  and  procedures 
(TTPs)  or  concepts  of  operations  (CONOPSs) — is  the  fastest,  but  not  necessarily  the  most 
effective,  response.  In  many  cases,  success  depends  on  the  introduction  of  new  equipment, 
technology,  or  weapon  systems. 

The  objective  of  this  paper  is  to  support  warfighters  in  the  achievement  of  success 
on  the  battlefield  by  enabling  the  DoD  to  assess  the  adaptability  of  current,  in-design,  and 
in-development  weapon  systems;  determine  how  modernization  upgrades  may  enhance  or 
degrade  adaptability;  and  design  future  weapon  systems  to  be  adaptable.  In  so  doing,  it 
seeks  to  offer  an  answer  to  the  question:  How  do  you  operationalize  adaptability  in  the 
DoD’s  technical  capability  base  and  its  capabilities  development  process,  and  measure  the 
degree  to  which  the  weapon  systems  resulting  from  those  processes  are  adaptable  (DSB, 
2011,  p.36)?3 

There  are  several  incentives  for  focusing  on  weapon  systems.  Unlike  other  potential 
sources  of  adaptability  (e.g.,  TTPs  and  CONOPSs),  systems  are  long-gestation,  long-lived 
assets  whose  design  constraints  prevail  for  decades.  And  these  assets  are  costly — 
Research,  Development,  Test,  &  Evaluation  (RDT&E)  and  procurement  accounts  combined 
are  approximately  one-third  of  the  DoD’s  budget  ($170  billion  in  2013).  Weapon  systems  are 
analytically  tractable  and  amenable  to  rigorous  examination  and  assessment,  as  they  are 
subject  to  physical  laws.  Such  analyses  and  assessments  could  serve  as  valuable  inputs 
into  strategies  for  developing  adaptive  TTPs,  CONOPSs,  skills,  and  organizations.  For 
example,  exposing  operators  to  unutilized  technical  capabilities  in  current  systems  could 
encourage  creative  uses  of  the  same.4  Additionally,  an  assessment  of  current  and  in- 


3  The  DSB  recommended  that  development  and  acquisition  planning  include  adaptability  as  a  specific 
requirement  metric. 

4  How  many  of  us  understand  the  technical  capabilities  of  our  smartphones?  If  more  did,  it  is 
reasonable  to  expect  that  heretofore  unknown  novel  uses  would  be  identified.  Consider  the 
extraordinary  number  and  types  of  apps  that  have  been  developed  by  the  iPhone  and  Android  user 
communities,  for  example. 
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development  systems  that  finds  a  lack  of  adaptability  might  suggest  that  a  cost-effective 
investment  strategy  for  achieving  adaptability  now  may  lie  in  those  other  arenas.5 

This  paper  presents  a  set  of  concepts,  working  definitions,  a  framework,  and  a 
quantitative  approach  for  evaluating  adaptability  in  current,  in-design,  and  in-development 
weapon  systems  and  for  supporting  dynamic  trade  space  analyses  to  enable  the  design  of 
adaptive  future  systems.6  It  proceeds  with  a  discussion  of  three  distinct  but  related 
concepts:  responsiveness,  flexibility,  and  adaptability. 

Concepts  and  Working  Definitions 

These  concepts  are  not  new  to  the  physical  systems  analytical  community.  Their 
discussion  here,  however,  is  novel  in  that  the  lens  through  which  they  are  considered  is  that 
of  the  defense  of  the  nation.  The  concepts  of  responsiveness,  flexibility,  and  adaptability  are 
taken  from  the  dynamic  system  and  control  theory  fields  and  modified  for  use  by  the  DoD. 

•  Adaptability  is  a  measure  of  the  change  in  the  state  variable  of  interest. 

•  Flexibility  is  a  measure  of  the  effort  required  to  transition  from  state  x0  to  x^7 
It  is  inversely  related  (or  negatively  correlated)  to  the  effort  required  to 
transition  to  a  new  state.  A  system  that  is  flexible  requires  less  effort  to  be 
reconfigured  to  reach  state  x^ 

•  Responsiveness  is  a  measure  of  the  time  required  to  transition  from  state  x0 
to  x-i.  Responsiveness  is  inversely  related  (or  negatively  correlated)  to  the 
time  required.  A  system  that  is  responsive  requires  less  time  to  transition 
between  states. 

Considering  these  concepts  within  the  context  of  the  paper’s  objective,  working 
definitions  for  assessing  against  and  designing  to  adaptability  are  as  follows: 

•  Adaptability  is  a  measure  of  the  potential  set  of  missions  (or  possible  states 
within  a  mission  space)  that  can  be  supported.8 

•  Flexibility  is  an  inverse  measure  of  the  costs  of  adapting  (effort,  capability 
tradeoffs,  and  dollar  costs);  the  greater  the  costs  to  adapt,  the  less  flexible 
the  weapon  system. 

•  Responsiveness  is  an  inverse  measure  of  the  time  required  to  adapt  (i.e., 
transition  within  a  mission  space  or  between  missions). 

These  definitions  are  distinct  but  related  and  apply  equally  well  to  weapon  systems 
and  their  physical  subsystems.  The  acquisition  community  will  likely  see  a  relationship 


5  For  a  study  on  skills  development,  see  Burns  and  Freeman  (2010).  Alternative  assessment 
approaches  might  be  more  appropriate  for  alternative  acquisition  strategies.  Other  strategies  could  be 
grounded  in  procuring  larger  quantities  of  single-purpose  platforms  or  based  on  a  systems-of-systems 
approach  to  capability  development. 

6  The  Joint  Requirements  Oversight  Council  (JROC)  recently  sent  a  memorandum  to  all  DoD 
Components  and  Agencies  to  encourage  requests  for  Key  Performance  Parameter  (KPP)  relief  if 
KPPs  appear  out  of  line  with  cost-benefit  analysis.  A  dynamic  trade  space  analysis  methodology 
would  be  a  useful  tool  for  informing  such  requests.  See  Joint  Requirement  Oversight  Council  (2013). 

7  For  alternate  definitions,  see  Ferguson,  Siddiqi,  Lewis,  and  de  Week,  2007. 

8  For  a  discussion  of  possible  states  within  the  same  mission  space,  see  Conley  and  Tillman,  2012. 
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between  these  terms  and  the  traditional  acquisition  parlance  of  performance  ( potential ), 
dollar  cost,  and  schedule. 

Assessing  and  Designing  for  Adaptability 

Weapon  systems  and  platforms  typically  remain  in  service  for  long  periods,  during 
which  change  often  occurs — some  of  which  is  manageable  and  some  not.  Routinely 
dynamic  international,  operational,  and  fiscal  environments  should  encourage  the  DoD  to 
assess  the  adaptability  of  its  current  and  planned  weapon  systems  and  ensure  that  future 
systems  are  designed  to  facilitate  adaptation  to  changing  circumstances. 

Assessing  and  designing  for  adaptability  should  not  be  confused  with  doing  so  for 
robustness.9  Even  though  each  concept  refers  to  the  ability  of  a  system  to  handle  change, 
the  nature  of  the  change  as  well  as  the  system’s  reaction  to  it  in  each  case  is  very  different. 
Adaptability  implies  the  ability  of  a  design  to  satisfy  changing  requirements,  whereas 
robustness  involves  satisfying  a  fixed  set  of  requirements  despite  changes  in  the  system’s 
operating  environment  (Saleh,  Hastings,  &  Newman,  2003).  An  adaptable  design  is  an 
active  way  to  deal  with  future  mission  and/or  operating  environment  uncertainty,  as  it 
includes  core  design  resource  margins  assessed  as  most  likely  to  be  relevant  across  a  wide 
range  of  potential  futures.  This  approach  is  intended  to  minimize  risks  and  maximize 
opportunities.  Conversely,  a  robust  design  is  passive,  as  it  focuses  on  a  system  performing 
a  fixed  set  of  requirements  satisfactorily  regardless  of  the  future  environment  (de  Neufville  & 
Scholtes,  2011,  pp.  6,  39). 

Framework  for  Assessment  and  Design 

Designing  for  adaptability  requires  discussions — early  in  the  capability  development 
process — of  mission  requirements  (i.e.,  capabilities),  design  resources,  technical  limitations, 
operational  constraints,  dollar  costs,  and  their  coupling  to  physical  and  engineering 
relationships.  These  factors  comprise  a  high-order  framework  that  can  also  be  used  for 
assessing  the  adaptability  of  current  and  in-development  systems.  Why  these  factors? 
System  capabilities  (e.g.,  range,  speed,  payload,  force  protection,  probability  of  kill)  depend 
on  how  design  resources  (e.g.,  internal  volume,  weight,  power)  are  consumed  and  supplied 
by  physical  subsystems  (e.g.,  engine,  armor,  fuel)  and  operational  constraints  (e.g., 
transportability  weight  limit,  high  hot  limits)  and  are  further  bounded  by  fiscal  constraints. 
These  factors,  while  few  in  number,  comprehensively  describe  a  system  from  both  a  user 
and  technical  perspective.  Their  relationships  are  illustrated  in  Figure  1 . 


9  Designing  for  adaptability  should  also  not  be  confused  with  designing  for  an  incremental  acquisition 
approach  to  support  an  evolutionary  acquisition  (EA)  strategy.  In  EA,  a  fixed  requirement  is  met  over 
time  by  developing  several  increments,  each  dependent  on  available  mature  technology.  See 
Enclosure  2  of  OUSD(AT&L),  2008. 
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Fiscal  Constraints 


Figure  1.  Relationships  Comprising  the  Framework 

Capability  envelopes  and  adaptability  draw  from  the  same  reservoir,  (i.e.,  design 
resources  and  operational  constraints).  Consider,  as  an  example,  the  potential  adaptability 
and  flexibility  of  a  nominal  infantry  fighting  vehicle  (IFV)  initially  developed  to  support  a 
cross-country  terrain  mission.  The  measure  of  adaptability  will  be  the  number  of  potential 
missions  the  vehicle  could  support,  with  a  specific  focus  on  assessing  adaptability  for  urban 
operations.  The  measure  of  flexibility  will  be  the  dollar  costs  and  tolerability  of  capability 
trades  required  in  order  to  adapt. 

Because  this  nominal  vehicle  was  intended  to  traverse  quickly  across  wide-open 
terrain,  its  original  design  sacrificed  force  protection  for  speed  and  range.  Using  the  vehicle 
in  urban  operations  would  require  significantly  more  force  protection,  thus  requiring  up- 
armoring.  It  is  assumed  that  there  are  numerous  bolt-on  armor  kits  available  at  reasonable 
dollar  cost  that  would  satisfy  this  need;  however,  utilizing  such  kits  would,  in  turn,  consume 
additional  weight  and  power  design  resources.  That  consumption  would  then  result  in 
reduced  vehicle  speed  and  range  (capability  tradeoffs). 

The  vehicle  in  this  example  could  be  assessed  as  adaptable,  flexible,  and  responsive 
with  regard  to  urban  operations  missions: 

•  Adaptable:  the  vehicle  had  unutilized  design  resources  ( weight  and  power) 
that  enabled  up-armoring  to  provide  additional  force  protection  required  for  a 
new  mission  (urban  operations). 

•  Flexible:  the  dollar  cost  and  capability  tradeoff  cost  of  adapting — force 
protection  for  speed  and  range — were  reasonable  and  tolerable. 

•  Responsive:  applying  bolt-on  armor  is  not  a  time-intensive  activity. 

The  example  highlights  the  fact  that  assessing  adaptability  is  necessary,  but  not 
sufficient,  for  making  decisions  regarding  potential  system  modifications/reconfigurations  or 
initial  designs.  Flexibility  and  responsiveness  should  also  be  considered.  Note  that  when 
adaptability  requires  capability  tradeoffs,  it  should  not  necessarily  be  construed  as  negative, 
as  the  trades  may  be  considered  tolerable  or  even  desirable.  In  the  example,  the  loss  of 
speed  and  range  was  deemed  tolerable  given  the  urban  operating  environment. 

Focus  on  Design  Resources 

The  framework  suggests  that  design  resource  margins  are  the  appropriate  focus  for 
both  assessing  and  designing  for  adaptability.  Why  a  margins-based  approach  when  others 
have  argued  that  modularity  is  the  best  route  for  “buying”  adaptability?  The  focus  on 
resource  margins  was  not  motivated  by  analytical  or  engineering  preference;  rather,  it  was 
driven  by  current  defense  strategic  guidance  and  a  review  of  the  DoD’s  recent  capability 
development  and  acquisition  history. 
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Current  guidance  calls  for  developing  “cutting  edge”  technical  capabilities.  This  is  not 
new  guidance,  as  DoD  has  historically  developed  systems  with  the  objective  of  achieving 
superior  technical  performance.  But  its  implications  are  significant  from  an  engineering 
perspective.  Superior  technical  performance  comes  from  integral  designs,  not  modular 
ones.  There  is  wide  agreement  on  this  point  across  engineering  communities.  Modularity 
comes  with  technical  performance  costs;  it  tends  to  favor  “business  performance”  over 
technical  performance  (Holtta-Otto  &  de  Week,  2007;  Whitney,  2004).  It  is  not  surprising, 
then,  that  a  review  of  recent  MDAPs  (including  some  in  the  design  phase)  showed  an 
overwhelming  majority  of  the  programs  were/are  being  designed  as  highly  complex,  highly 
capable,  integrated-architecture  systems — for  example,  the  F-22,  F-35,  DDG-51  Flight  III, 
and  GCV. 

From  an  assessment  perspective,  then,  the  systems  populating  the  assessment 
sample  are  almost  entirely — if  not  entirely — integral  rather  than  modular.  From  a  design 
perspective,  since  it  is  assumed  that  the  objective  of  retaining  “cutting  edge”  capability  will 
not  be  relaxed  any  time  soon,  integral  designs  will  likely  persist.  Design  resource  margins 
are  the  most  appropriate  metric  for  measuring  adaptability  in  integral  systems  and, 
therefore,  are  the  focus  of  this  approach. 

With  all  of  that  being  said,  systems  can  certainly  be  designed  as  integral-modular 
hybrids.  Even  in  that  type  of  design,  however,  a  focus  on  design  resource  margins  is  most 
appropriate  for  assessing  or  embedding  adaptability.  It  is  instructive  to  consider  recent 
comments  on  the  subject  by  Chief  of  Naval  Operations  Admiral  Greenert  (2012).  In 
promoting  payload  modularity,  Greenert  argued  the  design  of  future  platforms  “must  take 
into  account  up  front  the  volume,  electrical  power,  cooling,  speed,  and  survivability  needed 
to  effectively  incorporate  new  payloads  throughout  their  service  lives”  (p.  4).  Stated 
differently,  the  platforms  must  be  designed  with  margins  sufficient  to  handle  future  payloads. 

The  remainder  of  this  paper  applies  the  concepts,  working  definitions,  and  framework 
introduced  above  to  the  tasks  of  assessing  the  adaptability  of  current  and  planned  weapon 
systems  and  supporting  dynamic  trade  space  analyses  to  enable  the  design  of  future 
adaptive  systems. 

Enhancing  or  Degrading  Adaptability 

As  mentioned  previously,  capabilities  and  adaptability  draw  from  the  same  reservoir 
of  design  resources,  and  those  resources  can  either  be  consumed  or  supplied  by  physical 
subsystems.  When  assessing  or  designing  for  adaptability,  uncertainty  should  be 
considered  on  the  supply  side  (e.g.,  the  state  or  trends  of  technology)  as  well  as  the 
demand  side  (e.g.,  the  operating  environment).  On  the  supply  side,  it  may  be  that  future 
technological  advancements  in  physical  subsystems  could  supply  future  design  resources  to 
current  platforms.  For  example,  lighter  armor  could  supply  weight  margin,  and  more  efficient 
batteries  could  supply  both  weight  and  internal  space  margins.  Considering  the  supply  side 
enables  assessments  of  the  contributions  that  system  upgrades  would  make  to  the 
adaptability  of  the  system.  Upgrades  that  consume  design  resources  degrade  future 
adaptability,  while  those  that  supply  resources  enhance  it. 

Proofs  of  Concept 

Assessing  the  adaptability,  flexibility,  and  responsiveness  of  current  and  in¬ 
development  systems  requires  an  understanding  of  mission  requirements,  key  design 
resources  and  their  utilization,  physical  subsystems,  operational  constraints,  costs,  and  their 
interactions  and  relationships.  In  this  section,  several  proofs  of  concept  are  offered  to 
illustrate  the  assessment  and  design  methodologies. 
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Designing  and  Dynamic  Trade  Space  Analysis:  Proofs  of  Concept 

The  approaches  to  designing  for  adaptability  and  supporting  dynamic  trade  space 
analysis  are  nearly  identical,  absent  the  first  item  listed  below: 

•  Decide  whether  the  system  will  be  developed  to  be  generally  or  specifically 
adaptable.  This  requires  explicit  recognition  of  the  level  of  uncertainty 
associated  with  the  missions  and/or  environments  in  which  the  system  is 
intended  to  operate. 

•  Identify  the  capabilities  desired  (and,  more  directly,  the  physical  subsystems 
that  will  provide  them)  and  the  associated  design  resources  that  are  either 
supplied  or  consumed  by  them. 

•  Develop  a  physics-based  understanding  of  the  interaction  between 
capabilities  desired,  physical  subsystems,  and  design  resources. 

•  Identify  operational  constraints  that  limit  performance. 

•  Identify  costs. 

In  this  section  of  the  paper,  a  nominal  IFV  will  be  used  to  present  two  proofs-of- 
concept.  The  first  example  will  demonstrate  how  adaptability  can  be  rigorously  considered  in 
the  design  of  a  system.  It  will  also  highlight  an  important  issue  not  yet  addressed  in  our 
design  discussion — strategic  value  versus  tactical  cost.  The  second  example  will  illustrate  a 
more  complex  dynamic  trade  space  analysis.  These  proofs-of-concept  offer  stark  examples 
of  how  adaptability  and  capability  draw  from  the  same  reservoir  (i.e.,  design  resources  and 
operational  constraints).  Table  1  details  basic  performance  and  technical  assumptions  that 
will  be  used  in  both  proofs.  The  cells  labeled  “Trade  space”  in  the  Capabilities  (Desired) 
column  will  be  the  focus  of  the  dynamic  trade  space  analysis. 


Table  1.  Nominal  IFV  Performance  and  Technical  Assumptions 
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Designing  for  Specific  Adaptability:  Force  Protection 

This  proof  explores  potential  vehicle  designs  that  could  enable  future  increases  in 
ballistic  force  protection,  thereby  ensuring  the  IFV  will  remain  operationally  effective  in 
increased-threat  environments.  It  is  assumed  that  a  number  of  alternative  futures  have  been 
assessed,  resulting  in  a  bounded  range  of  potential  force  protection  requirements — 
STANAG  Level  4  to  STANAG  Level  5. 

For  any  potential  design  considered  in  this  proof,  the  performance  objectives  listed  in 
Table  1  (e.g.,  mobility  and  reliability)  must  not  be  compromised  if/when  future  upgrades  to 
the  vehicle  occur.  A  design  that  supports  adaptability  to  increase  passive  armor  in  the  future 
must  ensure  now  that  the  weight  design  resource  is  properly  calibrated  and  supplied  to 
enable  this  future  addition.  The  primary  physical  subsystems  that  supply  the  weight 
resource  are  suspension  and  structure  (see  the  Full  Spectrum  row  in  Table  1).  Weight  also 
interacts  with  the  mobility  requirement  and  drives  the  engine  size. 

Referring  back  to  the  bulleted  items  that  constitute  the  approach  to  designing  for 
adaptability,  the  first  three  have  been  satisfied:  specific  adaptability  was  selected;  desired 
capabilities  and  their  associated  physical  subsystems  and  design  resources  were  identified; 
and  the  interactions  between  them  were  understood.  The  remaining  two  items  are 
addressed  as  follows:  it  is  assumed  that  the  C-17  will  remain  the  heavy  airlift  vehicle  for  the 
foreseeable  future;  therefore,  the  transportability  weight  limit  of  the  C-17  will  be  considered 
an  operational  (and,  therefore,  design)  constraint.  Regarding  cost  assumptions,  see  Patel 
and  Fischerkeller  (2013). 

Two  vehicle  designs  were  considered,  to  illustrate  the  relationships  between  their 
relative  adaptability,  flexibility,  and  responsiveness.  One  (“Optimized  Vehicle”)  represents  a 
vehicle  designed  optimally  to  support  the  lower  bound  force  protection  requirement — 
STANAG  4 — with  no  margin  incorporated  for  bolt-on  armor  upgrades  to  increase  the  force 
protection  level.  The  other  (“Adaptable  Vehicle”)  represents  a  vehicle  designed  (with  regard 
to  suspension  and  structure)  to  supply  the  maximum  possible  weight  design  margin  to 
support  the  addition  of  future  force  protection  capability;  in  effect,  it  was  designed  to  support 
bolt-on  steel  armor  upgrades  to  increase  force  protection  to  the  upper  bound  force 
protection  requirement — STANAG  5.  Table  2  shows  the  comparisons. 
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Table  2.  Performance  and  Relative  100th  Unit  Procurement  Costs  ($K  of 
BY2012) — Optimized  vs.  Adaptable  Designs 
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The  performance  columns  in  Table  2  show  that  both  vehicles  perform  equally  well  up 
through  an  operating  environment  requiring  a  force  protection  level  of  STANAG  4  +  60% 
STANAG  5.  They  do  so,  however,  through  very  different  means.  While  both  vehicles  carry 
steel  armor  at  STANAG  4,  the  Optimized  Vehicle’s  force  protection  capability  is  increased  by 
replacing  steel  with  titanium  armor.  This  must  be  a  zero-sum  weight  exchange  because  the 
optimized  vehicle  was  not  designed  to  carry  additional  weight.  Conversely,  the  Adaptable 
Vehicle  was  designed  to  carry  additional  weight  and  has  its  force  protection  capability 
increased  through  additional  bolt-on  steel  armor.  At  STANAG  4  +  70%  STANAG  5,  the 
maximum  weight  the  Optimized  Vehicle  can  carry  is  exceeded,  resulting  in  system  failure. 
This  is  not  the  case  for  the  Adaptable  Vehicle.  Not  only  can  it  still  operate  effectively  in  that 
environment,  it  can  also  accommodate  additional  bolt-on  steel  armor  to  operate  effectively 
up  to  STANAG  5. 

Flexibility  is  captured  in  the  chart  via  the  relative  (A)  cost  columns.  At  STANAG  4,  the 
Optimized  Vehicle  has  a  lower  relative  unit  procurement  cost,  however,  as  requirements 
increase,  costs  increase  sharply  relative  to  the  Adaptable  Vehicle  because  more  expensive 
titanium  armor  is  needed  to  maintain  desired  mobility  and  reliability.  Embedding  adaptability 
made  for  a  more  flexible  vehicle,  as  its  upgrade  costs  are  less  sensitive  to  changes  in 
requirements. 

Finally,  inferred  but  not  captured  directly  in  this  chart  is  responsiveness.  Steel  armor 
must  be  stripped  before  titanium  armor  is  applied  to  the  Optimized  Vehicle.  This  is  far  more 
time-intensive  than  bolting  on  steel  to  the  Adaptable  Vehicle.  The  Adaptable  Vehicle,  then, 
is  more  responsive. 

Designing  for  General  Adaptability:  Dynamic  Trade  Space  Analysis 

This  general  adaptability  proof  illustrates  a  far-wider  range  of  possible  system 
adaptations  and  their  dependencies.  The  technical  and  cost  assumptions  presented  for  the 
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nominal  IFV  (Table  1)  will  again  be  used  in  this  proof.  This  analysis  will  assume  that  an 
adaptable  IFV  is  designed  with  a  20%  weight  margin,  100%  electrical  power  margin,  and  a 
33%  power  margin  relative  to  the  optimized  design,  to  support  future  unspecified  capabilities 
for  currently  unknown  missions  and  operating  environments.  Weight  and  power  were 
selected  because  they  dominate  the  design,  as  can  be  seen  in  their  relevance  to  nearly 
every  capability  desired  in  Table  1.  Power,  in  particular,  was  selected  because  experience 
tells  that  it  can  be  traded  in  the  future  to  support  many  different  types  of  capabilities  either 
directly  or  indirectly.  As  such,  it  is  a  core  design  resource  that  supports  adaptability  to  many 
potential  futures.  As  before,  the  performance  objectives  highlighted  in  Table  1  (e.g.,  mobility, 
reliability,  and  transportability)  must  not  be  compromised  in  any  potential  design. 

In  order  to  illustrate  one  iteration  of  a  dynamic  trade  space  analysis,  Figure  2  shows 
the  cost,  force  protection,  number  of  dismounts  carried,  and  urban  accessibility  (percent  of 
urban  areas  accessible)  trade  space  for  a  vehicle  designed  with  a  20%  weight  margin.  This 
is  a  high-order  analysis,  a  level  at  which  adaptable  design  analyses  should  commence.  The 
models  behind  this  analysis  are  typically  called  screening  models  and  represent  simple, 
transparent,  and  readily  understandable  representations  of  the  physical  interactions  of  the 
physical  subsystems.10  Screening  models  allow  numerous  iterations,  to  consider  potential 
adaptable  designs  relatively  quickly.  They  provide  the  ability  to  explore  the  art  of  the 
possible  with  minimal  expense  (time  and  dollars).  The  time  for  more  complex,  engineering 
point  models  is  later  in  the  design  phase,  not  sooner  (de  Neufville  &  Scholtes,  201 1 ). 

This  dynamic  trade  space  analysis  illustrates  a  number  of  opportunities  for 
consumption  of  that  20%  weight  margin  in  the  future.  For  example,  high  urban  accessibility 
would  come  at  the  cost  of  squad  size  and  force  protection. 


10  The  Institute  for  Defense  Analyses  has  created  a  suite  of  screening  models  for  GCV  analysis.  They 
were  the  basis  for  analyses  presented  in  Figure  1 . 
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Figure  2.  Dynamic  Trade  Space  Analysis  Supported  by  General  Adaptability 

Design 

Additional  high-order  analyses  are  also  possible.  Perhaps  the  100%  electrical  power 
margin  could  be  used  for  additional  sensors  and  electronics.  Would  that  affect  internal 
volume  available  for  dismounts?  Would  that  additional  weight  consumption  constrain  future 
armor  choices?  Should  mobility  or  transportability  be  traded?  And  so  on.  The  multitude  of 
questions  one  could  ask  is,  again,  a  strong  motivation  for  using  these  low-resolution 
analytical  tools  iteratively  at  the  outset  of  the  design  process. 

Strategic  Value  Versus  Tactical  Cost 

The  above  analysis  introduces  an  important  aspect  of  designing  for  adaptability — 
strategic  value  versus  tactical  cost  (i.e.,  nominal  program  costs).  Equating  the  two, 
especially  when  planning  for  an  uncertain  environment,  is  a  mistake.  While  the  relative  costs 
of  the  Optimized  Vehicle  at  STANAG  4  are  less,  should  future  emergent  threats  demand 
higher  force  protection,  the  costs  of  up-armoring  (and  concomitant  capability  tradeoffs) 
arguably  decrease  its  strategic  value  compared  to  that  of  the  Adaptable  Vehicle.11 

As  with  insurance,  the  strategic  value  of  a  system  should  be  assessed  in  terms  of  its 
contributions  over  all  possible  futures.  Insurance  and  adaptability  are  justified  by  the  value 


11  Our  example  assumed  a  smooth  design  and  development  process.  Often,  however,  requirements 
are  changed  post-Milestone  B,  which  leads  to  cost  growth.  This  cost  is  not  considered  in  the 
example.  In  reality,  then,  it  may  very  well  be  that  tactical  costs  for  optimized  and  adaptable  platforms 
are  often  comparable  as  changes  in  requirements  could  more  easily  be  addressed  by  adaptable 
designs  (see  GAO,  2011,  pp.  14-15;  Bolten  etal.,  2008). 
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they  bring  when  relevant  events  occur,  not  by  their  continual  use  (de  Neufville  &  Scholtes, 
201 1 ,  p.  1 1 ).  If  we  consider  a  “relevant  event”  as  a  future  circumstance  that  requires  the 
specification  of  new  system  requirements,  several  such  events  inevitably  occur  over  the 
service  lives  of  systems  as  new  technologies  or  new  threats  emerge.  At  the  right  price,  we 
willingly  buy  insurance  as  a  hedge  against  uncertain  future  events.  So,  too  should  DoD  as  it 
faces  an  uncertain  future.  But  how  can  decision-makers  determine  whether  the  price  for 
adaptability  is  reasonable?  Figure  3  illustrates  a  decision  support  chart  that  was  constructed 
using  the  optimized  and  adaptable  vehicle  cost  data  presented  in  Table  2. 

Selecting  either  an  adaptable  or  optimized  system  is  a  “bet”  on  future  trends  rather 
than  any  one  specific  outcome.  For  this  example,  selecting  adaptability  is  a  “bet”  that  future 
adversaries  will  employ  capabilities  that  would  require  significantly  more  force  protection 
than  is  required  in  current  systems.  Conversely,  selecting  an  optimized  design  is  a  “bet”  that 
future  adversaries  will  not  employ  capabilities  that  would  require  significant  changes  to 
current  force  protection  levels. 
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Select  adaptable  system 


Select  optimized  system 


Select  adaptable  system  if  you  believe  that  there 
is  an  X%  chance  that  this  range  of  protection 
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Figure  3.  Capability  Development  and  Acquisition  Decision  Support  Chart 

The  following  examples,  constructed  from  referencing  Figure  3,  illustrate  how  the 
chart  can  quantitatively  inform  capability  development  and  acquisition  decisions. 

Specifically,  we  can  describe  the  “bet”  that  leadership  is  making  in  more  quantitative  and 
rigorous  terms. 

An  adaptable  system  provides  the  greatest  strategic  value  if: 

•  Leadership  is  confident  there  is  at  least  a  small  chance  that  adversaries  will 
employ  capabilities  that  would  require  force  protection  levels  above  STANAG 

4.1,  or 

•  The  weight  margin  can  be  utilized  for  other  emergent  requirements. 

An  optimized  system  provides  the  greatest  strategic  value  if: 

•  Leadership  is  confident  that  there  is  a  high  chance  that  adversaries  will  not 
employ  capabilities  that  would  require  force  protection  levels  above  STANAG 

4.1. 
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Costs  from  Table  2  are  embedded  in  Figure  3  via  a  present  value  (PV)  analysis  of 
the  optimized  and  adaptable  systems.  The  Confidence  Level  contours  (color  code) 
represent  the  minimum  annualized  probability  at  which  the  adaptable  system  provides  more 
value  (e.g.,  lower  PV). 

The  approach  taken  to  create  Figure  3  can  be  replicated  to  create  similar  capability 
development  and  acquisition  support  tools  for  other  systems.  It  enables  decision-makers  to 
explicitly  account  for  uncertainty  in  their  choices  and  review  the  consequences  of  that 
accounting.  While  preferably  brought  to  bear  sooner,  such  an  approach  would  be  very 
beneficial  at  the  Analysis  of  Alternatives  decision  point. 

Which  Resource  Margins  and  How  Much? 

Effective  implementation  of  a  margin-based  approach  to  designing  adaptability  into 
weapon  systems  requires  choosing  which  design  resources  should  be  allocated  margin  (or 
not)  and  calculating  the  size  of  that  margin  such  that  additional  system  value  in  future 
uncertain  environments  could  be  realized  by  consuming  (or  supplying)  them  in  those 
environments. 

The  designing-for-adaptability  process  presented  previously  informs  resource  margin 
decisions.  In  the  proofs-of-concept,  the  capabilities  were  fixed  values  and  the  type  and  value 
of  margin  were  known  (the  design  resource  of  weight  with  the  percentage  of  20).  In  actual 
dynamic  trade  space  analysis,  all  should  be  considered  potential  variables  whose  values 
(and  also  types,  in  the  case  of  margins)  would  be  determined  for  a  final  design  through 
numerous  exploratory  analyses.  Numerous  iterations  allow  the  analysts,  operators,  and 
other  stakeholders  opportunities  to  consider  many  different  approaches  to  a  design  that 
satisfies  known  requirements  and  enables  adaptability  for  unknown  future  requirements. 

The  creative  value  of  multiple  iterations  cannot  be  overstated  and  again,  highlights  the 
importance  of  using  low-resolution  screening  models  early  in  the  design  process. 

As  trade  space  within  and  across  capabilities  and  margins  is  being  explored,  Key 
Performance  Parameters  (KPPs)  grounded  in  long-term  forecasts  in  which  confidence  is 
moderate  to  low  should  be  considered  first  for  trade  as  the  design  team  seeks  to  embed  a 
margin  for  potential  future  requirements.  One  need  only  perform  a  cursory  review  of  a 
handful  of  System  Threat  Assessment  Reports  (STARs)  to  see  several  examples  of 
moderate  and  low  confidences  being  cited.  Returning  to  a  point  made  earlier,  routine 
failures  to  accurately  forecast  futures  should  engender  modesty.  That  modesty  can  be 
operationalized  as  design  margins  to  increase  the  potential  strategic  value  of  a  platform.  A 
similar  perspective  could  be  taken  when  reviewing  KPP  threshold  (required)  and  objective 
(desired)  values.  To  the  degree  the  differences  in  those  values  are  based  on  different  levels 
of  confidence  in  near-  vs.  long-term  forecasts,  that  delta  should  be  considered  trade 
space — plan  for  the  relative  certainty,  prepare  for  the  uncertainty. 

This  approach  can  and  should,  where  appropriate,  be  complemented  by  experience. 
For  example,  the  Navy  incorporates  power  margins  on  ships  as  part  of  their  service  life 
allowances  based  largely  on  historical  experience.  Similarly,  based  on  mission  experience, 
the  National  Aeronautics  and  Space  Administration  (NASA)  incorporates  into  all  flight 
systems  a  10%  margin  for  power  and  5°  C  thermal  design  margin  to  respond  to  post-launch 
uncertainties  associated  with  the  mission  and  environment,  respectively  (NASA,  2009,  pp. 
13,82). 

Conclusion 

Adaptability,  flexibility,  responsiveness — these  terms  need  not  be  empty  descriptors 
of  the  force  desired  by  the  White  House  and  the  DoD.  They  can  be  operationalized  as 
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metrics  against  which  the  force  can  be  assessed  and  towards  which  it  can  be  designed. 
Current  operational  and  fiscal  realities  call  for  an  approach  to  enable  those  efforts.  Absent 
one,  the  DoD  risks  stumbling  forward  into  an  uncertain  strategic  and  operational  future, 
possibly  making  significant  force  structure,  modernization,  and  future  weapon  system  design 
decisions  that,  at  a  minimum,  do  nothing  to  enhance  the  force’s  adaptability  and  could,  quite 
possibly,  facilitate  its  degradation. 

A  general  utilization  assessment  of  the  current  force’s  major  systems’  design 
margins  would  offer  insights  into  the  potential  for  adaptability  to  emergent  circumstances  in 
an  uncertain  future  environment.  A  more  focused  look  at  those  margins  deemed  most 
relevant  to  future  missions  and  operating  environments  in  which  high  confidence  exists  also 
would  yield  valuable  and  actionable  insights. 

Designs  for  incremental  modernization  programs  or  entirely  new  weapon  systems, 
which  are  expected  to  be  in  the  field  for  decades,  should  explicitly  incorporate  adaptability. 
When  considering  upgrades  or  new  designs,  the  perspective  of  strategic  value  vs.  tactical 
cost  should  rule  the  day.  It  was  noted  previously  that  the  DSB  recommended  an  adaptability 
requirement  for  all  future  systems.  The  DoD  enterprise  is  populated  by  systems  engineers, 
operators,  and  other  stakeholders  who  are  both  intelligent  and  fallible;  consequently, 
unanticipated  threats  and  opportunities  often  emerge  late  in  the  course  of  development 
(post-Milestone  B)  and  long  after  initial  fielding.  But  changes  in  requirements  need  not  be  as 
cost-imposing  as  they  often  are;  adaptable  designs  could  provide  opportunities  to  apply 
those  costs  toward  achieving  greater  strategic  system  value  by  enabling  systems  to  be 
modified  to  execute  currently  unknown  missions  and  operate  in  currently  unknown 
environments.  Where  uncertainty  is  abundant,  an  adaptability  requirement  should  be  non- 
negotiable — it  must  be  a  “need-to-have,”  not  a  “nice-to-have.” 

Preparing  for  an  uncertain  future  is  not  an  insurmountable  challenge  for  the  DoD. 
Significant  RDT&E  and  procurement  decisions  that  take  adaptability  into  account  can  be 
informed  by  rigorous  analyses  and  assessments.  We  hope  this  paper  has  offered  useful 
concepts,  working  definitions,  and  approaches  to  inform  an  intelligent  path  forward  that 
enables  the  DoD  to  prepare  to  be  wrong. 
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Abstract 

The  U.S.  Defense  community  denotes  an  ecosystem  of  system  or  software  component 
producers,  system  integrators,  and  customer  organizations.  For  a  variety  of  reasons  this 
community  now  embraces  the  need  to  utilize  open  source  software  (OSS)  and  proprietary 
closed  source  software  (CSS)  in  the  system  capabilities  or  software  components  it  acquires, 
design,  develops,  deploys,  and  sustains.  But  the  long-term  transition  to  agile  and  adaptive 
capabilities  that  integrate  bespoke  or  legacy,  OSS  and  CSS  components,  has  surfaced  a 
number  of  issues  that  require  acquisition-research-led  approaches  and  solutions.  In  this 
paper,  we  identify  and  describe  six  key  issues  now  found  in  the  Defense  software  ecosystem: 
(1 )  unknown  or  unclear  software  architectural  representations;  (2)  how  to  best  deal  with 
diverse,  heterogeneous  software  IP  licenses;  (3)  how  to  address  cybersecurity  requirements; 
(4)  challenges  arising  in  software  integration  and  release  pipelines;  (5)  how  OSS  evolution 
patterns  transform  software  IP  and  cybersecurity  requirements;  and  (6)  the  emergence  of 
new  business  models  for  software  distribution,  cost  accounting,  and  software  distribution.  We 
use  the  domain  of  command  and  control  systems  under  different  acquisition  scenarios  as  our 
focus  to  help  illuminate  these  issues  along  the  way.  We  close  with  suggestions  for  how  to 
resolve  them. 

Introduction 

The  U.S.  Defense  community,  which  includes  the  military  services  and  civilian- 
staffed  agencies,  is  among  the  world’s  largest  acquirers  of  commodity  and  bespoke 
(custom)  software  systems.  The  Defense  community  further  extends  its  reach  and  influence 
on  a  global  basis  through  national  treaties  and  international  alliances  through  enterprises 
like  NATO.  The  Department  of  Defense  (DoD),  other  government  agencies,  and  most  large- 
scale  business  enterprises  continually  seek  new  ways  to  improve  the  functional  capabilities 
of  their  software-intensive  systems  while  lowering  acquisition  costs.  The  acquisition  of  open 
architecture  (OA)  systems  that  can  adapt  and  evolve  through  replacement  of  functionally 
similar  software  components  is  an  innovation  that  can  lead  to  lower  cost  systems  with  more 
powerful  functional  capabilities.  OA  system  acquisition,  development,  and  deployment  are 
thus  seen  as  an  approach  to  realizing  Better  Buying  Power  (BPP)  goals  for  lowering  system 
costs,  achieving  technical  excellence,  enabling  innovation,  and  advancing  the  acquisition 
workforce  (Kendall,  2015). 
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Bespoke  software  systems  are  produced  and  integrated  within  the  Defense 
community.  In  addition  Defense  system  acquisition  or  procurement  enterprises  also  obtain 
wares  from  most  non-Defense  industry  providers  of  software  systems,  applications,  or 
services  (i.e.,  the  mainstream  software  products  or  services  industry).  The  acquisitions  often 
entail  software  procurement  or  development  contracts  valued  in  the  millions  to  hundreds-of- 
millions  of  dollars  (Myers  &  Obendorf,  2001 ).  At  this  scale  of  endeavor  and  economic  value, 
certain  kinds  of  software  engineering  (SE)  research  problems  arise  that  are  not  visible  or  are 
insignificant  in  smaller  scale  SE  R&D  efforts. 

In  this  paper,  we  focus  attention  to  the  slice  of  this  world  that  focuses  on  the 
development  and  deployment  of  software-intensive  command,  control,  communication, 
cyber  and  business  systems  (hereafter,  C3CB).  We  further  limit  our  focus  to  the  most 
general  software  elements  found  in  C3CB  system  capabilities;  for  example,  software 
infrastructure  components,  common  development  technologies  supporting  app/widget 
development,  and  mission-specific  apps/widgets,  in  particular  widgets  produced  with  the 
Ozone  Widget  Framework  (Conley  et  al.,  2014).  OWF  (now  called  the  Ozone  Platform  or 
OZP)  was  initially  developed  by  the  NSA,  though  is  now  identified  as  Government  OSS 
(GOSS)  and  supported  by  a  third-party  contractor.  It  is  widely  used  within  the  Defense  and 
Intelligence  community.  The  growing  importance  of  OZP  within  the  Defense  community  has 
directed  focus  to  the  production  and  integration  of  C3CB  system  capabilities  to  be 
assembled  using  it.  This  focus  drives  open  discussion  of  and  broad  exposure  to  emerging 
research  issues  that  arise  from  the  production  and  integration  (or  software  engineering — 

SE)  of  software  components,  and  these  in  turn  raise  challenges  for  acquisition  management 
and  personnel.  Specifically,  we  draw  attention  to  issues  surrounding  the  development, 
integration,  and  deployment  of  multi-version  and  multi-variant  software  systems  composed 
from  various  open  source  software  (OSS)  and  proprietary  (CSS)  software  elements  or 
remote  services  (Scacchi,  2002,  2010),  eventually  including  recent  efforts  to  support  Web- 
compatible  services  and/or  mobile  devices  in  C3CB.  This  focus  also  provides  exposure  to 
future  C3CB  system  capabilities  composed  from  apps  acquired  through  various  acquisition 
regimes,  including  apps  downloaded  from  different  Defense  community  app  stores  (George, 
Morris,  O’Neil,  et  al.,  2013;  George  et  al.,  2014). 

Recent  Scenarios  for  Acquisition  of  OA  Software  Capabilities 

Interest  in  open  source  software  (OSS)  within  the  U.S.  Department  of  Defense  (DoD) 
and  military  services  first  appeared  more  than  10  years  ago  (Bollinger,  2003;  Scacchi  & 
Alspaugh,  2008).  More  recently,  it  has  become  clear  that  the  U.S.  Defense  community  has 
committed  to  a  strategy  of  acquiring  software-intensive  systems  across  the  board  that 
require  or  utilize  an  “open  architecture”  (OA)  which  may  incorporate  OSS  technology  or  OSS 
development  processes  that  can  help  Defense  customer  organizations  to  achieve  better 
buying  power  (Kendall,  2015).  Why?  Among  the  reasons  identified  is  the  desire  to  realize 
more  choices  among  software  component  producers  or  integrators,  as  producers  and 
integrators  often  act  in  ways  that  lock  their  customer  organizations  into  overly  costly  and 
sometimes  underperforming  and  difficult  to  sustain  systems.  One  approach  being  explored 
focuses  attention  to  agile  and  adaptive  OA  software  components  that  are  acquired  and 
assembled  (integrated)  as  C3CB  system  capabilities  (assembled  capabilities  or  AC)  that  are 
acquired  and  shared  by  multiple  parties  via  independent  “lines  of  efforts”  acting  within  an 
ecosystem  of  producers,  integrators,  and  consumer  organizations  (Reed  et  al.,  2014; 
Scacchi  &  Alspaugh,  2015).  The  goals  of  the  AC  approach  include  a  shorter  delivery  and 
update  cycle  for  mission  components  and  an  improved  cybersecurity  posture.  We  explain 
this  approach  as  follows. 
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The  AC  approach  contemplates  independent  acquisition  lines  of  effort  for  different 
types  of  OA  software  components  that  can  be  acquired  from  independent  providers: 


•  Mission  Components  enable  C3CB  processes  and  present  common 
operating  picture  data  to  end-users.  Mission  components  may  be  realized  as 
apps/widgets  that  may  be  deployed  on  mission-specific  platforms,  including 
those  operating  on  secured  Web/mobile  devices. 

•  Common  Development  Technology  provides  AC  development  tools  and 
common  run-time  applications  servers  that  support  the  mission  components. 
The  servers  are  bundled  with  Shared  Infrastructure,  as  follows. 

•  Shared  Infrastructure  Components  combine  local/remote  application  servers 
and  data  repositories  with  networking  services  and  platforms. 


Assembled  capabilities  therefore  represent  alternative  configurations  of  mission- 
specific  components  that  are  produced  with  common  development  technology  for 
deployment  on  shared  infrastructure  technology  platforms. 

Independent  Lines  of  Effort  (LOEs)  by  single  or  multi-party  acquisition  for  mission 
components,  common  development  technologies,  or  shared  infrastructure  components,  are 
expected  to  greatly  accelerate  development  and  fielded  deployment.  This  acceleration 
entails  tradeoffs  in  increased  dependency  and  risk  management.  Independent  LOEs  enable 
at  least  three  alternative  scenarios  for  acquiring  OA  C3CB  system  capabilities. 

1 .  Use  current  strategy  and  acquisition  capabilities.  Here  there  is  no  focus  on 
AC  that  utilizes  mission  components,  common  development  technologies,  or 
shared  infrastructure  components. 

2.  Augment  deployed  systems  with  mission  components  and  common 
technologies.  Augmentation  is  either  for  (a)  new  mission  functionality;  (b) 
modernization  “in  place”  so  that  part  of  the  original  system  is  deprecated  as 
the  new  mission  components  are  delivered;  or  (c)  infrastructure  replacement 
over  parts  of  original  system  that  may  be  combined  with  modernization 
efforts. 

3.  Focus  efforts  on  production,  integration,  security  assurance,  and  deployment 
of  mission  components  that  use  common  technologies  and  shared 
infrastructure,  and  that  can  be  assembled  into  different  ACs.  This  can  entail 
production,  integration,  and  delivery  of  all  mission  components  in  one 
contract  vehicle;  or  alternatively,  the  delivery  of  mission  components 
partitioned  across  multiple  acquisition  contract  vehicles,  so  as  to  spread  and 
manage  risk,  while  insuring  multi-party  buy-in  commitment. 

The  following  efforts  provide  examples  where  these  alternative  C3CB  acquisition 
scenarios  can  be  considered.  First,  the  Air  Force’s  Theater  Battle  Management  Core 
System-Force  Level  (TBMCS-FL),  which  manages  air  tasking  orders  and  airspace 
management,  among  other  things,  is  being  harvested  for  current  operational  capabilities. 
These  capabilities  can  then  be  encapsulated  and  delivered  as  mission  components  for  other 
C3CB  systems,  using  OZP  widgets  and  supporting  common  technologies.  The  C2AOS  C2IS 
acquisition  scenario  also  intends  to  deliver  harvested  functionality  as  mission  components. 
Air  Force  AOC  (Air  Operations  Center)  is  planning  to  include  C2AOS  C2IS  as  the 
replacement  for  TBMCS-FL,  and  will  use  the  Navy  ACS  (hence  indicating  the  need  for  multi¬ 
party  acquisition  agreements).  This  in  turn  implies  the  need  for  Joint  C2,  and  needs  to  be 
copied  to  all  Services.  It  represents  an  opportunity  to  reduce  duplicate  activities  for 
producing  equivalent  C3CB  system  capabilities.  Second,  the  Army’s  Distributed  Common 
Ground  System  (DCGS-A)  currently  uses  mission  components  for  visualization  (over  300 
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widgets  available).  DCGS-A  will  incorporate  metadata  mission  components  that  utilize  the 
DCGS  Integration  Backbone  (DIB).  Third  and  last,  the  Navy  is  deploying  CANES  and  ACS 
(Agile  Core  Services)  shared  infrastructure  to  its  fleet  as  a  modernization  effort  (Guertin, 
Sweeney,  &  Schmidt,  2015). 

There  are  now  a  number  of  policy  directives  within  the  Defense  community  that 
formally  recognize  that  OSS  system  elements  can  be  treated  as  commercial-off-the-shelf 
(COTS)  components,  and  that  bespoke  software  system  development  projects  will  utilize  an 
OA,  unless  otherwise  justified  and  approved.  Thus,  developing  contemporary  C3CB  that 
incorporate  both  OSS  and  new/legacy  CSS  elements  are  “business  as  usual.”  However, 
many  legacy  Defense  community  system  capability  producers  are  hesitant  about  how  best 
to  engineer  such  OA/OSS  systems.  For  example,  does  an  OA  system  imply/require  that  its 
software  architecture  be  explicitly  modeled,  be  accessible  for  sharing/reuse  (e.g.,  as  a 
Reference  Model),  and  be  modeled  in  a  form/notation  that  is  amenable  to  architectural 
analysis  and  computational  processing  (“Software  Architecture,”  2016)?  Therefore,  we  can 
begin  to  identify  what  kinds  of  SE  research  issues  can  be  observed  and  investigated  within 
the  Defense  community  associated  with  its  transition  to  OA  systems  and  OSS  software 
elements,  specifically  for  Web  and  Mobile  devices  within  the  realm  of  C3CB. 

OA,  Open  APIs,  OSS,  and  CSS 

OA  C3CB  system  capabilities  are  assembled  with  mission  components,  common 
development  technologies,  and  infrastructure.  Infrastructure  components  are  broadly 
construed  to  include  non-mission  specific  software  functionality  or  operations.  Such 
components  can  include  computer  operating  systems,  Web  servers,  database  management 
systems,  cloud  services,  mobile  device  management  middleware,  and  others,  along  with 
desktop,  mobile,  or  smartphone-based  Web  browsers,  word  processors,  email  and 
calendaring,  text/voice  chat,  and  end-user  media  players.  Example  infrastructure 
components  include  the  U.S.  Army’s  Common  Operating  Environment  (COE),  the  Navy’s 
Consolidated  Afloat  Networks  and  Enterprises  Services  (CANES)  Afloat  Core  Services 
(ACS)  (Guertin,  Sweeney,  &  Schmidt,  2015),  and  similar  elements  in  the  Joint  Intelligence 
Environment. 

Common  development  technologies  are  common  software  development  tools, 
libraries,  or  frameworks  used  to  implement  the  necessary  software  functionality  so  that  new 
or  legacy  mission  components  can  be  integrated  into  mission-specific  software  capabilities. 
Software  technology  frameworks  (or  common  implementation  libraries)  like  Oracle  Java  8, 
Ozone  Platform,  OpenJDK  (OSS  Java  Development  Kernel  for  Android  app  development), 
and  the  NASA  World  Wind  Java  SDK;  programming  languages  like  Java  or  C++;  and 
scripting  languages  like  Javascript  may  be  utilized  as  common  development  technologies 
for  developing  mission  components.  Other  software  production  capabilities  like  the  Navy 
Tactical  Cloud  and  CANES  integrate  both  infrastructure  and  common  development  tools  like 
Hadoop,  MapReduce,  and  other  mission  data  analysis  tools  for  the  Tactical  Cloud,  and  the 
Agile  Core  Services  and  Java  for  CANES. 

Mission  components  represent  a  hybrid  assortment  of  (a)  simple  widgets — small,  thin 
apps  similar  in  spirit  to  those  acquired  and  downloaded  from  online  app  stores  (like  a  clock, 
calculator,  dictionary,  sticky  note,  or  unit  converter);  (b)  singular  widgets — more  substantial 
functional  components  either  created  new  (bespoke)  or  extracted  from  legacy  systems  that 
must  run  on  a  specific  local  computing  platform  (e.g.,  shipboard  fire  control  system);  or  (c) 
compound  widgets — hosted  in  a  cloud  and  run  as  a  remote  cloud  service  over  a 
single/multi-tiered  client-server  software  architecture  (e.g.,  Google  Maps,  NASA  World 
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Wind),  and  thus  potentially  accessible  and  usable  on  a  Web/mobile  computing  platform 
(Google  Chrome  Web  browser  running  on  a  secure  Android  mobile  device). 

OA  seems  to  simply  suggest  software  system  architectures  incorporating  OSS/CSS 
infrastructure,  common  development  technologies,  and  mission  components  that  all  utilize 
open  application  program  interfaces  (APIs).  But  not  all  software  system  architectures 
incorporating  OSS/CSS  components  and  open  APIs  will  produce  an  OA,  since  whether  an 
architecture  is  an  OA  depends  on  (a)  how/why  OSS/CSS  and  open  APIs  are  located  within 
it;  (b)  how  OSS/CSS  and  open  APIs  are  implemented,  embedded,  or  interconnected  within 
it;  (c)  whether  the  copyright  (Intellectual  Property)  licenses  assigned  to  different  OSS/CSS 
components  encumber  all/part  of  the  architecture  into  which  they  are  integrated;  and  (d) 
choices  among  alternative  architectural  configurations  and  APIs  that  may  or  may  not 
produce  an  OA  (cf.  Scacchi  &  Alspaugh,  2008).  This  can  lead  to  situations  in  which 
acquisition  contracts  stipulate  a  software-intensive  system  with  an  OA  and  OSS/CSS 
components,  but  the  resulting  software  system  may  or  may  not  embody  an  OA.  This  can 
occur  when  the  architectural  design  of  a  system  constrains  the  system  requirements:  if  not 
all  requirements  can  be  satisfied  by  a  given  system  architecture,  if  requirements  stipulate 
specific  types  or  instances  of  OSS/CSS  (e.g.,  Web  browsers,  content  management  servers), 
if  an  architecture  style  (Bass,  Clements,  &  Kazman,  2003)  is  implied  by  given  system 
requirements,  or  if  requirements  are  implied  by  the  choice  to  incorporate  legacy  software 
capabilities  with  one  architectural  style  that  are  to  be  wrapped  within  mission-specific 
widgets  with  a  different  architectural  style. 

Application  domain  of  interest:  C3CB  with  Web/Mobile  Devices  Utilizing  Widgets 
C3CB  are  common  information  system  applications  that  support  modern  military  operations 
at  a  regional,  national,  or  global  level.  These  applications  may  be  focused  to  address 
common  military  mission  planning,  mapping,  resource  status  tracking  and  scheduling, 
mission  performance,  and  monitoring  activities  through  application  sub-systems.  However, 
closely  related  C3CB  systems  applications  are  also  in  common  use  within  civilian/public 
safety  agencies,  public  infrastructure/utility  operations,  live  television  and  sports  event 
broadcasting,  massively  multi-player  online  game  operations  centers,  and  even  in 
international  motorsports  racing  competition  events  like  Formula  1 .  So  the  study  of  software 
production  and  system  integration  issues  arising  in  the  Defense  community  can  inform 
awareness  of  similar  issues  in  other  non-Defense  software  system  domains,  and  vice  versa. 

Modern  C3CB  applications  are  increasingly  expected/planned  to  be  composed  from 
best-available  software  components,  whether  OSS  or  CSS,  utilizing  bespoke  or  legacy 
software  capabilities.  Furthermore,  as  smartphones,  tablets  and  laptop  computers  are  being 
brought  into  the  workplace,  so  too  is  interest  increasing  within  the  Defense  community  in 
supporting  the  acquisition  and  development  of  Web-compatible  widgets  and  mobile  apps, 
provided  through  an  emerging  ecosystem  of  component  producers  and  system  integrators, 
for  configuration  into  secure  OA  C3CB  software  system  capabilities  (George  et  al.,  2014; 
Reed  et  al.,  2012;  Reed  et  al.,  2014;  Scacchi  &  Alspaugh,  2013a;  Scacchi  &Alspaugh, 

2015).  Common  software  elements  for  such  systems  include  Web  browsers  open  to 
extensions  like  custom  mission-specific  Map  widgets,  and  remote  content  servers,  email  and 
calendaring,  word  processing,  local/networked  file  servers,  and  operating  systems.  The  data 
processed  by  the  software  may  be  of  high-relevance  to  military  missions/operations,  or  may 
just  be  the  daily  grind  of  data  manipulated  by  “productivity”  applications  which  most  of  us 
use  routinely  to  perform/enact  our  work  assignments.  Security  has  been  mostly  addressed 
through  system  isolation  or  “air  gaps”  to  the  outside  world  due,  for  example,  to  airborne  or 
afloat  capability  deployments.  But  this  is  no  longer  common  practice,  and  cybersecurity 
concerns  have  risen  to  the  top  of  functional  and  non-functional  requirements  for  all  such 
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C3CB  applications.  New  OA  systems  are  now  required  to  be  secure  by  design,  by 
implementation,  and  through  release,  deployment  and  evolution,  as  well  as  subject  to 
independent  testing  and  certification.  Secure  OA  designs  can  then  entail  different  schemes 
for  encapsulating  different  (sets  of)  components,  use  of  virtualization  schemes,  shims  and 
wrappers,  encrypting  data  transfers  and  storage,  and  configuring  multi-level  system  access 
capabilities.  But  we  have  found  examples  in  which  different  OA  system  designs  and 
configurations  propagate  security  obligations,  and  privacy  protections  and  access  rights  are 
either  mediated  or  nullified  by  different  software  component  IP  licenses  or  system  updates. 

OA  Ecosystems  Within  the  Defense  Community 

In  our  view,  a  software  ecosystem  is  a  network  of  software  component  producers, 
system  integrators,  and  customer  organizations.  In  the  Defense  community,  producers  and 
integrators  are  commonly  industrial  entities  (defense  contractors),  while  customer 
organization  are  military  program  offices.  Figure  1  presents  an  abstract  view  of  a  software 
ecosystem  that  associates  software  components  or  apps  with  their  producers,  system 
architectures  with  system  integrators,  and  delivered  component  or  integrated  application 
systems  with  their  customers.  We  also  add  annotations  to  indicate  that  each  component  or 
app  has  its  own  software  IP  license,  and  that  integrated  systems  delivered  to  customers 
come  with  some  composition  of  IP  license  obligations  and  rights  propagated  through  the 
system’s  OA. 


Figure  1.  An  Abstract  Software  Ecosystem  Rendered  as  a  Network  of  Software 
Component  Producers,  Integrators  of  Systems/AC,  and  End-User 
Consumer  Organizations 


mtSTANTIA  PER  SCIENTIAL 
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There  is  growing  interest  within  the  Defense  community  in  transitioning  to  acquiring 
complex  software  system  capabilities  via  an  agile  and  adaptive  ecosystem  (Reed  et  al., 
2012;  Reed  et  al.,  2014;  Scacchi  &  Alspaugh,  2015),  where  components  may  be  sourced 
from  alternative  producers  or  integrators,  allowing  for  more  competition,  and  ideally  lowering 
costs  and  improving  the  quality  of  software  elements  that  arise  from  a  competitive 
marketplace  (Kendall,  2015).  But  this  adaptive  agility  to  mix,  match,  reuse,  mashup,  swap, 
or  reconfigure  integrated  systems,  or  to  accommodate  end-user  architecting  (Garlan  et  al., 
2012)  as  in-house  integrations  of  mission  components,  requires  that  systems  be  compatible 
with  or  designed  to  utilize  an  OA.  Consequently,  we  can  identify  six  kinds  of  emerging 
research  challenges  or  issues  for  software  capability  acquisition  that  we  have  observed 
within  the  U.S.  Defense  community  as  they  move  to  produce,  integrate,  deploy  and  evolve 
OA  systems  for  C3CB  system  capabilities  that  utilize  contemporary  OSS  and 
bespoke/legacy  CSS  components.  These  issues  center  around  (1)  unclear  representations 
of  OA  software  system  capabilities,  (2)  how  best  to  accommodate  diverse  intellectual 
property  licenses  when  combining  bespoke/legacy  OSS/CSS  mission  components,  (3)  how 
to  accommodate  diverse  and  complicated  cybersecurity  requirements,  (4)  technical 
challenges  arising  from  alternative  ways  to  integrate  and  deploy  diverse  software 
components,  (5)  how  to  accommodate  many  different  paths  within  the  Defense  community 
that  drive  software  component  evolution,  and  (6)  how  to  estimate  and  manage  the  costs  of 
acquiring,  deploying,  and  sustaining  diverse  software-based  mission  components  and  C3CB 
system  capabilities.  These  are  examined  in  the  next  section. 

With  this  background  and  sets  of  concepts  for  understanding  a  simplified  view  of  the 
world  of  C3CB  software  systems,  we  now  turn  to  identify  and  examine  a  set  of  issues  that 
are  now  recurring  in  the  acquisition,  design,  development,  and  deployment  of  such  systems. 

Emerging  Issues  in  Developing  and  Deploying  OA  C3CB  Systems  Within 
Different  Acquisition  Scenarios 

There  are  at  least  six  kinds  of  emerging  research  challenges  or  issues  for  software 
capability  acquisition  that  we  have  observed  within  the  U.S.  Defense  community  as  it  moves 
to  OA  systems  for  C3CB  system  capabilities. 

Unknown  or  Unclear  OA  Solutions 

An  OA  entails  a  documented  representation  of  software  capability  described  in  an 
architectural  description  language  that  specifies  component  types,  component 
interconnections  and  connector  types,  open  APIs,  and  their  properties  and 
interrelationships.  The  common  core  of  a  C3CB  system  OA  resembles  most  enterprise 
business  systems,  as  C3CB  are  a  kind  of  management  information  system  for  navigating, 
mapping,  tracking  resources;  scheduling  people  and  other  resources;  producing  plans  and 
documentation;  and  supporting  online  email,  voice,  or  video  communications.  Figure  2 
depicts  an  OA  representation  that  can  also  serve  as  a  “reference  model”  for  a  C3CB 
software  product  line  (Womble  et  al.,  201 1 ).  Figure  3  further  expands  the  sub-architecture  of 
software  components  that  denote  configurations  of  mission-specific  components  as  widgets. 
Thus,  C3CB  system  capabilities  can  compose  or  reuse  multiple  or  nested  OA  reference 
models. 
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Figure  2.  OA  Reference  Model  for  Common  Software  Component  Types 

Note.  This  is  an  OA  reference  model  for  common  software  component  types  including 
widgets  interconnected  within  integrated  C3CB  system  capability.  Components  come  from 
producers  that  are  assembled  into  OA  C3CB  capabilities  by  system  integrators. 
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Figure  3.  OA  Reference  Model  for  Common  Types  of  Software  Widget 

Components 

Note.  This  figure  is  an  OA  reference  model  for  common  types  of  software  widget  components 
that  can  be  connected  and  integrated  to  realize  mission-specific  C3CB  system  capabilities, 
within  the  overall  OA  shown  on  the  left-side  in  Figure  2.  Servers  may  be  secured  Web 
content  servers,  app  servers,  databases,  or  file  system  servers/repositories. 

The  next  piece  of  the  OA  challenge  we  are  studying  is  the  envisioned  transition  with 
the  Defense  community  to  C3CB  system  capabilities  being  composed  by  end-user  system 
integration  architects  (Garlan  et  al.,  2012)  working  within/for  customer  organizations,  or 
potentially  extended  by  end-users  deployed  in  the  field.  This  is  the  concept  that  surrounds 
the  transition  to  discovering  software  components,  apps,  or  widgets  in  Defense  customer 
organization  app  stores  (George  et  al.,  2013;  George  et  al.,  2014).  These  app  stores  are 
modeled  after  those  used  in  distributing  and  acquiring  software  apps  for  Web-based  or 
mobile  devices,  operated  by  Apple,  Google,  Microsoft,  and  others.  How  the  availability  of 
such  Defense  mission  capability  app  stores  will  transform  the  way  C3CB  systems  are 
produced,  or  even  if  legacy  Defense  industry  contractors  will  produce  them,  remains  to  be 
seen.  Said  differently,  how  app  stores  transform  OA  software  ecosystem  networks,  business 
models,  and  cybersecurity  practices  is  an  emerging  challenge  for  acquisition  and  SE 
research  in  the  Defense  community. 

Another  kind  of  challenge  arises  when  acquiring  new  or  retrofitting  legacy  C2 
software  system  applications  that  lack  an  open  or  explicit  architectural  representation 
identifying  major  components,  interfaces,  interconnections  and  remote  services  (if  any). 
Though  OA  reference  models  and  architectural  description  languages  are  in  use  within  the 
SE  research  community,  contemporary  C3CB  generally  lack  such  descriptions  or 
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representations  that  are  open,  sharable,  or  reusable.  This  may  be  the  result  of  legacy 
business  practices  in  the  Defense  community  that  see  detailed  software  architecture 
representations  as  proprietary  IP  rather  than  as  open,  sharable  technical  data,  even  when 
OSS  components  are  included  or  when  applications  sub-systems  are  entirely  made  of  OSS 
code.  An  alternative  explanation  reveals  that  complex  software  systems  like  common  Web 
browsers  (Mozilla  Firefox,  Google  Chrome,  Apple  Safari,  Microsoft  Internet  Explorer)  have 
complex  architectures  that  integrate  millions  of  SLOC  that  are  not  well  understood,  and  that 
entail  dozens  of  independently-developed  software  elements  with  complex  APIs  and  IP 
licenses  that  shift  across  versions  (Scacchi  &  Alspaugh,  2012).  For  such  systems  the  effort 
to  produce  an  explicit  OA  reference  model  is  itself  a  daunting  architectural  discovery, 
component/sub-system  extraction,  restructuring/refactoring,  and  continuous  software 
evolution  task  (Choi  &  Scacchi,  1990;  Kazman  &  Carriere,  1998).  Thus,  new  ways  and 
means  for  extracting  software  components  interconnections  and  interfaces  and  transforming 
them  into  higher-level  architectural  representations  of  mission-specific  apps/widget 
configurations  are  needed. 

Harvesting  legacy  source/executable  binary  code  entails  many  software  engineering 
challenges  that  constrain  acquisition  efforts.  First,  legacy  code  provides  too  much  technical 
detail  and  comparatively  little  abstraction  of  overall  system  configuration,  composition, 
components  and  interconnection/dependencies.  Second,  incongruent  computational  system 
models  (e.g.,  legacy  data-flow  versus  publish-subscribe  widgets)  or  hybrid  OA  AC  arise 
when  transitioning  legacy  system  software  elements  into  new  widget-based  mission 
components.  Third,  there  is  a  general  inability  to  visualize  or  analyze  (test,  selectively 
execute,  translate  into  another  programming  language,  etc.)  overall  system  configurations, 
interconnections,  or  interfaces.  Fourth,  lacking  these  three,  the  potential  for  general  software 
reuse  is  limited  to  executable  code  reuse,  which  is  the  lowest  common  denominator  for 
reuse.  Such  reuse  results  in  substantial  blocks  of  unused  code  that  cannot  be  easily 
removed  due  to  indiscernible  interdependencies.  Last,  when  configuring  mission 
components  that  entail  legacy  C2  software  applications  wrapped  for  integration  as  widgets, 
different  architectural  styles  can  inadvertently  be  mixed  (e.g.,  dataflow  architecture  for 
legacy  C2  software,  and  publish-subscribe  architecture  for  configured  mission  widgets), 
which  in  turns  raises  the  potential  for  architectural  mismatches  (Velasco-Elizondo  et  al., 
2013)  that  may  be  difficult  to  determine  or  detect  during  system  integration,  especially  when 
such  integration  activities  are  performed  by  end-user/consumer  organizations. 

Heterogeneously  Licensed  OA  Software  Capabilities 

OSS  components  are  subject  to  widely  varying  copyright,  end-user  license 
agreements,  digital  civil  rights,  or  other  IP  protections.  The  Open  Source  Institute  recognizes 
dozens  of  OSS  licenses  are  in  use,  though  the  top  10  represents  more  than  90%  of  the 
open  source  ecosystem  (Scacchi  &  Alspaugh,  2012).  This  is  especially  true  for  OSS 
components  or  application  systems  that  incorporate  source  code  from  multiple,  independent 
OSS  development  projects,  such  as  found  in  contemporary  Web  browsers  like  Firefox  and 
Chrome  which  incorporate  components  from  dozens  of  OSS  projects,  most  with  diverse 
licenses  (Scacchi  &  Alspaugh,  2012).  This  means  that  C3CB  system  capabilities  that  entail 
configuration  of  OSS/CSS  components  are  subject  to  complex  software  IP  obligations  and 
rights  that  may  defy  tracking,  or  entail  contradictory  legal  obligations  or  rights  (Alspaugh, 
Scacchi  &  Asuncion,  2010).  Determining  overall  IP  obligations  for  such  systems  is  generally 
beyond  the  scope  of  expertise  for  software  developers,  as  well  as  most  corporate  lawyers. 
Furthermore,  we  have  observed  many  ways  in  which  IP  licenses  interact  within  an  OA 
software  system,  such  that  different  architectural  design  choices  that  configure  the  same  set 
of  software  components  result  in  different  overall  system  obligations  and  rights. 
Understanding  multiple  license  interaction  and  IP  mismatches  is  far  too  confusing  for  most 
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acquisition  professionals  and  Program  Office  decision-makers  and  a  source  of  legal 
expense,  or  alternatively  expensive  indemnification  insurance  policies  by  the  software 
producers  or  system  integrators. 

One  complication  that  can  be  anticipated  here  arises  when  component  types  are 
replaced  with  versioned  component  instance  alternatives  (Scacchi  &  Alspaugh  2012). 
Consider  the  situation  where  a  Web  Browser  (e.g.,  Firefox  40.0.3  or  Chrome  47.0.2526. Ill 
(64-bit);  etc.)  component  has  a  specific  IP  license  (e.g.,  Mozilla  Public  License  2.0  or  GPL 
3.0)  associated  with  the  versioned  instance,  which  in  turn  may  be  viewed  by  system 
integrators  as  enabling/limiting  an  integrated  system’s  architectural  design,  depending  on 
how  different  components  are  interconnected  in  ways  that  may  or  may  not  propagate  (un-) 
desirable  IP  obligations  and  rights — a  concern  that  arises  frequently  when  using 
components  subject  to  the  GPL  (Scacchi  &  Alspaugh,  2008).  As  we  have  learned  in 
practice,  corporate  lawyers  employed  by  Defense  contractors  or  in  government  agencies  do 
not  have  solutions  for  how  to  resolve  such  complexities,  except  via  costly  overall  liability 
indemnification  schemes,  and  efforts  to  distribute  integrated  systems  with  many  IP 
obligations  and  few  rights  that  effectively  make  an  integrated  open  source  system  closed. 
This  in  turn  can  defeat  the  potential  opportunities  and  benefits  for  commitment  to  OA 
systems  that  integrate  OSS  components. 

Bespoke/legacy  software  components  for  OA  AC  design,  integration  and  delivery 
within  widgets  will  be  subject  to  their  bespoke/legacy  IP  obligations.  This  may  include  limits 
on  the  right  to  extract,  restructure,  or  reengineer  their  architecture  (cf.  Choi  &  Scacchi,  1990; 
Kazman  &  Carriere,  1998)  into  open  source  formats.  Similarly,  IP  licenses  associated  with 
OSS  or  new  CSS  components  may  impinge  on  their  integration  with  these  legacy 
components,  or  may  limit  disclosure  of  their  interfaces  that  would  allow  more  open 
integration  of  alternative  software  AC  configurations  developed  by  different  Defense 
community  component  producers  (Scacchi  &  Alspaugh,  2012). 

Nonetheless,  in  our  view,  OA  software  ecosystems  are  defined,  delimited,  and 
populated  with  niches  that  locate  specific  integrated  system  solutions  (Scacchi  &  Alspaugh, 
2012).  Furthermore,  we  see  that  these  niches  effectively  have  virtual  IP  licenses  that  must 
be  calculated  via  the  obligations  and  rights  that  propagated  across  integrated  system 
component  licenses  via  union,  intersection,  and  subsumption  relations  among  them 
(Alspaugh  &  Scacchi,  2012).  Such  calculation  may  appear  to  be  daunting,  and  thus  begs  for 
a  simpler,  tractable,  and  computationally  enforced  scheme  that  can  scale  to  large  systems 
composed  from  many  components,  as  well  as  be  practically  usable  by  C3CB  system 
capability  producers,  integrators,  and  acquisition  professionals.  In  such  a  scheme, 

OSS/CSS  licenses  could  formalize  IP  obligations  as  operational  requirements  (i.e., 
computationally  enforceable,  at  the  integrated  system  level)  instantiated  by  system 
integration  architects  (Alspaugh,  Scacchi,  &  Asuncion,  2010;  Alspaugh  &  Scacchi,  2013). 
Similarly,  customer/user  rights  are  then  non-functional  requirements  that  can  be  realized 
and  validated  as  access/update  capabilities  propagated  across  the  integrated  system 
(Alspaugh  &  Scacchi,  2013). 

Cybersecurity  for  OA  Software  Capabilities 

Cybersecurity  is  a  high  priority  requirement  in  all  C3CB  systems,  applications,  AC, 
and  platforms  (Scacchi  &  Alspaugh,  2013c;  Scacchi  &  Alspaugh,  2013d).  No  longer  is 
cybersecurity  something  to  be  addressed  after  C3CB  systems  are  developed  and 
deployed — cybersecurity  must  be  included  throughout  the  design,  development, 
deployment,  and  evolution  of  C3CB.  However,  the  best  ways  and  means  for  addressing 
cybersecurity  requirements  are  unclear,  and  oftentimes  somewhat  at  odds  with  one  another 
depending  on  whether  cybersecurity  capability  designs  are  specific  to  a  C3CB  platform 
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(e.g.,  operating  system  or  processor  virtualization;  utilization  of  low-level  operating  system 
access  control  or  capability  mechanisms);  component  producer  (secure  programming 
practices  and  verification  testing);  system  integrator  (e.g.,  via  use  secure  data 
communications  protocols  and  data  encryption);  customer  deployment  setting  (mobile: 
airborne  or  afloat;  fixed:  offices,  briefing  rooms,  operations  centers);  end-user  authentication 
mechanisms;  or  acquisition  policy  (e.g.,  reliance  on  third-party  audit,  certification,  and 
assurance  of  system  cybersecurity).  However,  in  reviewing  these  different  arenas  for 
cybersecurity,  we  have  found  that  the  cybersecurity  requirements  or  capabilities  can  be 
expressed  in  much  the  same  way  as  IP  licenses:  using  concise,  testable  formal  expressions 
of  obligations  and  rights.  Some  examples  follow  (capital  letters  are  placeholders  that  denote 
specified  system,  service,  or  component  contexts): 

•  The  obligation  that  a  user  must  verify  his/her  authority  by  password  or  other 
specified  authentication  process. 

•  The  obligation  that  all  components  connected  to  specified  component  C  must 
grant  it  the  capability  to  read  and  update  data  in  compartment  T. 

•  The  obligation  to  reconfigure  a  system  in  response  to  detected  threats,  when 
given  the  right  to  select  and  include  different  component  versions,  or 
executable  component  variants. 

•  The  right  that  a  user  or  software  component  may  read  and  update  data  in 
compartment  T  using  the  licensed  component. 

•  The  right  that  may  allow  replacement  of  a  specified  component  C  with  some 
other  vetted  component. 

These  examples  show  how  cybersecurity  requirements  can  be  expressed  or 
paraphrased  in  restricted  natural  language  (e.g.,  using  a  domain-specific  language)  into 
composite  specifications  that  denote  “security  licenses”  (Alspaugh,  Scacchi  &  Asuncion, 
2010;  Alspaugh  &  Scacchi,  2012).  In  this  way,  it  should  be  possible  to  develop  new  software 
analysis  tools  whose  purpose  is  to  interpret  cybersecurity  obligations  as  operational 
constraints  (executable)  or  provided  capabilities  (access  control  or  update  privileges), 
through  mechanisms  analogous  to  those  used  for  analyzing  software  licenses  (Alspaugh, 
Scacchi  &  Asuncion,  2010;  Alspaugh  &  Scacchi,  2012),  and  show  how  component  or  sub- 
system-specific  obligations  and  rights  can  be  propagated  across  a  system’s  architecture. 

We  similarly  envision  the  ability  for  OA  system  capabilities  to  be  produced  and 
integrated  according  to  different  cybersecurity  requirements,  depending  on  where  and  how 
they  are  deployed  (Scacchi  &  Alspaugh,  2013d).  For  example,  in  Figure  4  we  show  one 
possible  layout  of  software  components  that  confines  different  sub-configurations  within 
different  virtual  machines.  These  virtual  machines  may  also  be  hierarchically  nested,  as  is 
the  case  when  mission-specific  widgets  that  entail  legacy  C2  applications  must  be  securely 
confined  at  run  time  in  order  to  access  remote  servers,  in  contrast  to  a  secured  Web 
browser  running  on  a  secured  mobile  device. 
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Figure  4.  A  Configuration  of  Security  Confinement  Vessels  that  Encapsulate 
Infrastructure  Software  Components  and  Mission-Specific  Widgets  for  the 

OA  Shown  in  Figures  2  and  3 

Last,  the  inclusion  of  OSS  or  new  CSS  components  within  future  OA  C3CB  software 
systems  or  AC  will  be  amenable  to  current  approaches  to  cybersecurity  assurance,  as  we 
have  outlined  before  (Scacchi  &  Alspaugh,  2013d).  Mission  components  can  be  assessed 
for  cybersecurity  characteristics,  and  assembled,  without  triggering  reaccreditation. 

Similarly,  evolutionary  support  for  field-deployed  AC  can  allow  rapid  substitution  of  mission 
components  that  enable  rapid,  agile  response  to  cybersecurity  issues  in  mission 
components.  However,  legacy  CSS  components  which  were  developed  and  deployed 
before  current  cybersecurity  assurance  challenges  will  need  to  rely  on  “air-gap”  interfaces  at 
deployment  time  that  may  be  vulnerable  to  aggressive  exploits  delivered  through  mobile 
devices. 

Consequently,  we  believe  that  cybersecurity  can  be  addressed  in  the  future  using 
explicit,  computational  OA  representations  that  are  attributed  with  both  IP  and  cybersecurity 
obligations  and  rights. 

Software  Component  Build,  Release,  Deployment  (BRD)  Processes 

C3CB  applications  represent  complex  software  systems  that  are  often  challenging  to 
produce,  especially  when  conceived  as  bespoke  systems.  To  no  surprise,  acquisition  of 
these  systems  often  requires  a  development  life  cycle  approach,  though  some  system 
elements  may  be  fully-formed  components  that  are  operational  as  packaged  software  (e.g., 
commercial  database  management  systems,  Web  browsers,  Web  servers,  user  interface 
development  kits/frameworks).  C3CB  development  is  rarely  clean-sheet  and  less  likely  to  be 
so  in  the  future.  As  a  result,  component-based  system  development  approaches  are 
expected  to  dominate,  thus  relegating  system  integrators  (or  even  end-users)  to  perform  any 
residual  source  code  development,  inter-app  integration  scripting,  or  intra-app  extension 
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script  development.  But  software  process  challenges  arise  along  the  way  (Scacchi  & 
Alspaugh,  2013b). 

First  is  again  the  issue  noted  earlier  of  whether  there  is  an  explicit,  open-source  OA 
design  representation,  preferably  one  that  is  not  just  a  diagram,  but  instead  is  expressed  in 
an  architectural  design  language.  With  only  a  diagram  or  less,  then  is  little  or  no  guidance 
for  how  to  determine  whether  a  resulting  software  implementation  is  verifiable  or  compliant 
with  its  OA  requirements  or  acquisition  policies,  such  as  provision  or  utilization  of 
standardized,  open  APIs  to  allow  increased  software  reuse,  selection  of  components  from 
alternative  producers,  or  post-deployment  extensions  (Kendall,  2015). 

Second  is  the  issue  arising  from  system  development  practices  based  on  utilization 
of  software  components,  integrated  sub-systems,  or  turnkey  application  packages.  These 
software  elements  come  with  their  own,  possibly  unknown  requirements  that  are 
nonetheless  believed  to  exist  and  be  knowable  with  additional  effort  (Alspaugh  &  Scacchi, 
2013).  They  also  come  with  either  OSS  code  or  CSS  executables,  along  with  their 
respective  APIs.  These  components  must  be  configured  to  align  with  the  OA  specification. 
Consequently,  software  tool  chains  or  workflow  automation  pipelines  are  utilized  to  build  and 
package  internal/external  executable,  version-controlled  software  releases.  We  have  found 
many  diverse  automated  software  process  pipelines  are  used  across  and  sometimes  within 
software  integration  activities  (Scacchi  &  Alspaugh,  2013b).  These  pipelines  take  in  OSS 
code  files,  dependent  libraries,  or  repositories  (e.g.,  GitHub)  and  build  executable  version 
instances  that  are  then  subjected  to  automated  testing  regimes  that  include  simple  “smoke 
tests”  and  extensive  regression  testing.  Successful  builds  eventually  turn  into  packaged 
releases  that  may  or  not  be  externally  distributed  and  deployed  as  read y-to-in stall 
executables.  While  this  all  seems  modest  and  tractable,  when  one  sees  the  dozens  of 
different  OSS  tools  used  in  different  combinations  across  different  target  platforms  it 
becomes  clear  that  what  is  simple  when  small  becomes  a  complex  SE  activity  when  the 
scale  of  deployment  increases. 

Another  complication,  which  is  now  beginning  to  be  recognized  within  and  across 
BRD  processes  and  process  automation  pipelines,  arises  in  determining  when  and  how 
different  BRD  tool  chain  versions/configurations  can  mediate  cybersecurity  requirements  in 
the  target  system  being  built.  We  have  seen  cases  in  which  software  builds  and  deployed 
releases  are  assumed  to  integrate  to  functionally  equivalent  CSS  components,  but  which 
are  then  not  included  in  releases  due  to  IP  restrictions.  We  have  also  observed  and  reported 
how  functionally  equivalent  variants  as  well  as  functionally  similar  versions  may  or  may  not 
be  produced  by  BRD  tool  chains,  either  by  choice  or  by  unintentional  consequence.  This,  in 
our  opinion,  gives  rise  to  the  need  for  explicit  open-source  models  of  BRD  process 
automation  pipelines  that  can  be  analyzed,  tested,  reused,  and  shared  to  determine  whether 
release  versions/variants  can  be  verified  and/or  validated  to  produce  equivalent/similar 
releases  that  preserve  prior  cybersecurity  obligations  and  usage  rights. 

Last,  mixing  new  OSS  and  CSS  components  with  legacy  apps  wrapped  within 
widgets  will  complicate  build  and  release  processes  and  obscure  deployment  processes. 
Legacy  apps  encapsulated  within  mission-specific  widgets  will  commonly  need  to 
dynamically  link  executable  binary  components,  which  in  turn  increases  the  challenges  in 
their  testing  and  cybersecurity  assurance,  both  during  development  and  during  field 
deployment.  In  order  to  mitigate  these  technical  challenges  while  enabling  more  agile 
software  component  system  integration,  multi-component  OA  configurations  should  explicitly 
declare  pre/post  conditions  on  acceptable  input/output  parameter  values,  along  with 
exceptional  values,  that  in  turn  can  be  independently  verified  or  validated. 
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Software  Component  Evolution  Practices  Transmitted  Across  the  OA 
Ecosystem 

Software  evolution  is  among  the  most-studied  of  SE  processes.  While  formerly 
labeled  as  “software  maintenance,”  a  profitable  activity  mediated  through  maintenance 
contracts  from  software  producers  to  customers,  the  experience  of  OSS  development 
projects  and  practices  suggest  a  transition  to  a  world  of  continuous  software  development — 
one  that  foreshadows  the  emergence  of  continuous  SE  processes,  or  software  life  cycles 
that  just  keep  cycling  until  interest  falters  or  spins  off  into  other  projects.  OSS  development 
projects  rely  on  OSS  tools  that  themselves  are  subject  to  ongoing  development, 
improvement,  and  extension,  as  are  the  software  platforms,  libraries,  code-sharing 
repositories,  and  end-user  applications  utilized  by  OSS  developers  to  support  their 
development  work.  Developers  entering,  progressing,  or  migrating  within/across  OSS 
projects  further  diversify  the  continuous  development  of  the  most  successful  and  widely 
used  OSS  components/apps.  This  dynamism  in  turn  produces  many  ways  for  OSS  systems 
or  OA  systems  that  incorporate  OSS  components  to  evolve. 

Figure  5  portrays  different  software  evolution  patterns,  paths,  and  practices  we  have 
observed  arising  with  new  C3CB  applications  (Scacchi  and  Alspaugh  2012).  Here  we  see 
paths  from  a  currently  deployed,  executable  system  release,  to  a  new  deployed  release — 
something  most  of  us  now  accept  as  routine  as  software  updates  are  propagated  across  the 
Internet  from  producers,  through  integrators,  to  customers  and  end-users. 


Figure  5.  Different  Paths  and  Mechanisms  Through  Which  OA  Software  Systems 

Can  Evolve 

(Scacchi  &  Alspaugh,  2012) 
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Integrated  OA  systems  can  evolve  through  upgrades  of  functionally  equivalent 
component  variants  (patches)  as  well  as  through  substitution  of  functionally  similar  software 
components  sourced  from  other  producers  or  integrators.  In  Figure  6,  we  show  a  generic 
situation  that  entails  identifying  how  an  OA  consistent  with  that  depicted  in  Figure  2  may 
accommodate  the  substitution  and  replacement  of  a  locally  installed  word  processor 
application  with  a  remote  Web-based  word  processing  software  services  (for  example, 
Google  Docs  or  Microsoft  Office  365).  This  capability  is  a  result  of  utilizing  an  OA  that 
constitutes  a  reference  model  aligned  with  a  vendor-neutral  software  product  line.  This  is 
also  a  capability  sought  by  customer  organizations,  and  sometimes  encouraged  by  software 
producers  to  accommodate  their  evolving  business  models  (discussed  below).  While  the  OA 
remains  constant,  the  location  of  the  component  has  moved  from  local  to  remote/virtual,  as 
has  its  evolutionary  path.  Similarly,  the  cybersecurity  of  the  local  versus  remote  component 
has  changed  in  ways  that  are  unclear,  and  entail  a  different,  evolved  assurance  scheme. 
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Figure  6.  Alternative  Configurations  of  Integrated  Instance  Releases  of 
Components  Consistent  With  the  OA  in  Figure  2  That  Are  Treated  as 
Functionally  Equivalent  by  Customer  Organizations 

(Scacchi  &  Alspaugh,  2012) 

Next,  any  common  development  technology  used  to  support  production  or 
integration  of  mission  components  with  shared  infrastructure  components  must  recognize 
that  these  technologies  and  components  are  all  subject  to  independent,  mostly  autonomous 
evolution  practices  within  the  Defense  community.  For  example,  OZP  is  currently 
undergoing  evolution,  including  its  migration  to  Java  8  sourced  by  Oracle,  and  this  move  will 
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may  disrupt  the  correct  operation  of  widgets  already  produced  using  Java  7  common 
development  technologies.  Similarly,  new  OSS  and  CSS  components  will  evolve  due  to 
practices  arising  in  the  competitive  marketplace,  while  legacy  mission  components  wrapped 
within  widgets  will  have  obscure  or  opaque  evolution  practices  that  are  locked  into  legacy 
Defense  community  component  providers.  Legacy  components  will  also  limit  how  their 
encapsulating  widgets  evolve,  potentially  due  to  architectural  mismatches  or  dependencies 
to  legacy  systems  that  are  no  longer  supported,  operational,  or  compatible  with  current 
platform  technologies  (Velasco-Elizondo  et  al.,  2013). 

Overall,  the  evolution  of  software  components,  component  licenses,  component 
interconnects  and  interconnections,  and  interconnected  component  or  AC  configurations  are 
now  issues  that  call  for  research  efforts  to  help  make  such  patterns,  paths,  and  practices 
more  transparent,  tractable,  manageable,  and  scalable  within  an  OA  software  ecosystem, 
as  well  as  customers  seeking  the  benefits  of  openness,  sharing,  and  reuse. 

New  Business  Models  for  Acquisition  of  Software  Components  and  Widgets 

The  last  issue  we  address  is  the  newest  in  this  set  of  six  for  consideration  for  new 
acquisition  research.  While  the  field  of  acquisition  research  and  practice  has  long  paid 
attention  to  software  economics,  the  challenges  of  software  cost  estimation  are  evolving  in 
light  of  new  business  models  being  put  into  practice  by  software  producers  and  system 
integrators.  In  the  past,  software  development  projects  were  often  managed  by  a  single 
contractor  responsible  for  both  software  production  and  system  integration.  Costs  could  be 
assessed  through  augmentation  to  internal  business  accounting  practices  (e.g.,  budgeting, 
staffing  workloads,  time-sheet  reports,  project  schedules,  etc.).  But  a  move  to  OA 
ecosystems  means  that  multiple  producers  can  participate,  and  OA  schemes  accommodate 
switching  among  providers  while  a  system  is  being  integrated,  deployed,  or  evolved  in  the 
field.  This  in  turn  coincides  with  new  ways  and  means  to  electronically  distribute  software 
updates,  components,  or  applications,  as  well  as  new  ways  to  charge  for  software.  OSS 
components  may  be  acquired  and  distributed  at  “no  cost,”  but  their  integration  and  evolution 
are  charged  as  service  subscription,  or  as  time-effort  billings. 

We  have  already  seen  other  alternatives  for  costing  or  charging  for  software  that 
include  franchising;  enterprise  licensing;  metered  usage;  advertising  supported; 
subscription;  free  component,  paid  service/support  fees;  federated  reciprocity  for  shared 
development;  collaborative  buying;  donation;  sponsorship;  free/open  source  software  (e.g., 
Government  OSS — GOSS);  and  others.  So  how  are  customer  organizations,  especially  in 
the  Defense  community  where  software  cost  estimation  practices  are  routine,  supposed  to 
estimate  the  development  or  sustaining  costs  of  the  software  components  or  integrated 
systems  they  acquire  and  evolve,  especially  when  an  OA  system  allows  for  producers 
whose  components  come  with  different  costing/billing  schemes?  This  is  an  open  problem  for 
both  acquisition  research  and  software  engineering  practice. 

Overall,  new  OSS  and  CSS  components  are  experiencing  a  rapid  diversification  of 
acquisition  cost  models  and  practices,  while  legacy  components  are  generally  tied  to  single¬ 
source  contractors  as  a  result  of  utilizing  legacy  components  as  a  cost-avoidance  practice. 
All  of  the  preceding  five  factors  further  obfuscate  how  to  estimate  or  measure  software 
component/AC  development  costs,  schedules,  or  time  to  delivery/usage.  So  acquisition 
costs  of  systems  that  mix  and  match  new  OSS  and  bespoke  CSS  components,  together 
with  legacy  CSS  components,  will  be  difficult  to  cost-estimate  or  cost-manage.  This  in  turn 
will  limit  the  efficacy  of  BBP  3.0  practices  for  such  systems. 
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Discussion  and  Conclusions 

Our  study  reported  in  this  paper  identifies  a  set  of  technical  issues  and  risks  that  can 
dilute  the  cost-effectiveness  of  Better  Buying  Power  efforts.  It  similarly  suggests  that  current 
acquisition  practices  aligned  with  BBP  can  also  give  rise  to  acquisition  management 
activities  that  can  dominate  and  overwhelm  the  costs  of  OA  system  development.  This 
adverse  condition  can  arise  through  app/widget  vetting,  new  software  business  models, 
opaque  and/or  underspecified  acquisition  management  processes,  and  the  evolving 
interactions  of  new  software  development  and  deployment  techniques.  Unless  proactive 
investment  in  acquisition  research  and  development  can  give  rise  to  worked  examples, 
open-source  models,  and  new  acquisition  management  system  technologies,  the  likelihood 
of  acquisition  management  dominating  agile  development  and  adaptive  deployment  of 
component-based  OA  C2  system  capabilities  is  unsettling. 

Our  research  identified  and  analyzed  how  new  software  component  technologies  like 
OSS  infrastructure  components,  common  development  technology  components,  and 
mission-specific  widgets  for  Web-based  and/or  mobile  devices,  along  with  their  intellectual 
property  (IP)  license  and  cybersecurity  requirements,  engineering  and  evolution  processes, 
and  cost  estimating  practices  interact  to  drive  down  (or  drive  up)  total  system  costs  across 
the  system  acquisition  life  cycle.  The  availability  of  such  new  scientific  knowledge  and 
technological  practices  can  give  rise  to  more  effective  expenditures  of  public  funds  and 
improve  the  effectiveness  of  future  software-intensive  systems  used  in  government  and 
industry.  Thus,  a  goal  of  this  paper  was  to  explore  new  ways  and  means  for  achieving  cost- 
sensitive  acquisition  of  OA  software  systems,  as  well  as  identifying  factors  that  can  further 
decrease  or  increase  the  costs  of  such  systems. 

We  identified  and  examined  six  areas  for  research  arising  at  the  intersection  of 
software  engineering  and  acquisition  that  now  confront  the  Defense  community  (and 
perhaps  other  industries  as  well).  These  six  issues  areas  include  (1 )  the  lack  of  architecture 
representations  and  schemes  for  discovering  or  specifying  OA  system  designs;  (2)  OA 
systems  that  integrate  components  or  applications  subject  to  diverse,  heterogeneous  IP 
licenses;  (3)  how  to  manage  the  cybersecurity  of  OA  systems  during  system  design, 
development,  and  deployment;  (4)  software  process  challenges  and  evolving  disruptions  in 
seemingly  mundane  process  automation  pipelines;  (5)  software  evolution  patterns,  path, 
and  practices  in  OA  ecosystems;  and  (6)  how  new  business  models  are  upending  software 
cost  estimation  practices  and  outcomes.  All  of  these  research  areas  are  readily 
approachable,  and  research  results  are  likely  to  have  significant  practical  value,  both  within 
the  Defense  community  and  beyond. 

These  issue  areas  were  investigated  and  addressed  in  the  domain  of  command, 
control,  communication,  cyber  and  business  systems  (C3CB).  We  believe  all  are  tractable, 
yet  dense  and  sufficient  for  deep  sustained  research  study,  as  well  as  for  applied  research 
in  search  of  near-term  to  mid-term  practical  results. 

In  related  work  (Scacchi  &  Alspaugh,  2015),  we  have  called  for  specific  R&D 
investments  into  the  development  of  open  source,  domain-specific  languages  for  specifying 
open  architecture  representations  (or  architectural  description  languages)  that  are 
formalizable  and  computational,  as  well  as  supporting  annotations  for  software  license 
obligations  and  rights.  While  ADLs  have  been  explored  in  the  SE  research  community,  the 
challenges  of  how  software  architectures  mediate  software  component  licenses  and  cyber 
security  requirements  are  an  open  issue,  with  practical  consequences.  Similarly,  ADL 
annotations  that  assign  costs  or  cost  models  in  line  with  new  software  business  models  are 
an  open  problem  area.  We  have  also  called  for  R&D  investment  in  new  SE  tools  or  support 
environments  who  purpose  is  to  provide  automated  analysis  and  support  of  OA  systems  IP 
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and  cybersecurity  obligations  and  rights,  as  new  requirements  for  industrial  practice  in  large- 
scale  software  acquisition,  design,  development,  deployment,  and  evolution.  Such 
environments  are  the  automated  tools  that  could  be  used  to  model,  specify,  and  analyze 
dynamically  configurable,  component-based  OA  software  systems  expressed  using  the 
open  source  architectural  representation  schemes  or  ADLs  noted  here. 

Our  research  identifies  and  analyzes  how  OA  CBC3  system  capabilities  can  utilize 
software  components  and  mission-specific  widgets,  with  diverse  IP  license  and 
cybersecurity  requirements,  and  how  new  software  business  models  can  interact  to  affect 
total  system  costs  across  the  system  acquisition  life  cycle.  The  availability  of  such  new 
scientific  knowledge  and  technological  practices  can  give  rise  to  more  effective  expenditures 
of  public  funds  and  improve  the  effectiveness  of  future  software-intensive  systems  used  in 
Defense  community,  as  well  as  elsewhere  within  government  and  industry.  Hopefully,  this 
paper  serves  to  help  throw  light  into  how  software  engineering  and  acquisition  research  can 
inform  and  add  benefit  to  software  practices  within  the  Defense  community  through  ways 
and  means  that  further  advance  Better  Buying  Power  opportunities  and  outcomes. 

References 

Alspaugh,  T.  A.,  &  Scacchi,  W.  (2012,  September).  Security  licensing.  In  Proceedings  of  the 
Fifth  International  Workshop  on  Requirements  Engineering  and  Law  (pp.  25-28). 

Alspaugh,  T.  A.,  &  Scacchi,  W.  (2013).  Ongoing  software  development  without  classical 
requirements.  In  Proceedings  of  the  21st  IEEE  International  Conference  of 
Requirements  Engineering  (pp.  165-1 74).  Rio  de  Janeiro,  Brazil. 

Alspaugh,  T.  A.,  Scacchi,  W.,  &  Asuncion,  H.  (2010).  Software  licenses  in  context:  The 
challenge  of  heterogeneously  licensed  systems.  Journal  of  the  Association  for 
Information  Systems,  71(11),  730-755. 

Bass,  L.,  Clements,  P.,  &  Kazman,  R.  (2003).  Software  architecture  in  practice  (2nd  ed.). 
New  York,  NY:  Addison-Wesley  Professional. 

Bollinger,  T.  (2003,  January  2).  Use  of  free  and  open-source  software  (FOSS)  in  the  U.S. 
Department  of  Defense.  MITRE. 

Choi,  S.  C.,  &  Scacchi,  W.  (1990).  Extracting  and  restructuring  the  design  of  large  systems. 
IEEE  Software,  7(1 ),  66-71 . 

Conley,  K.,  Brockman,  B.,  Diercks,  P.,  George,  A.,  Lam,  W.,  Lozano,  A.,  Painter,  R.  & 
Tolentino,  G.  (2014,  June).  Achieving  information  dominance:  Unleashing  the  Ozone 
Widget  Framework.  In  Proceedings  of  the  19th  International  Conference  of  Command 
and  Control  Research  &  Technology  Symposium  (ICCRTS;  Paper  109).  Arlington,  VA. 

Garlan,  D.,  Dwivedi,  V.,  Ruchkin,  I.,  &  Schmerl,  B.  (2012).  Foundations  and  tools  for  end- 
user  architecting.  In  D.  Garlan  and  R.  Calinescu  (Eds.,),  Large-scale  complex  IT 
systems.  Development,  operation  and  management,  Lecture  Notes  in  Computer 
Science  (pp.  157-182),  Springer,  7539. 

George,  A.,  Galdorisi,  G.,  Morris,  M.,  &  O’Neil,  M.  (2014,  June).  DoD  application  store: 
Enabling  C2  agility.  In  Proceedings  of  the  19th  International  Command  and  Control 
Research  and  Technology  Symposium  (Paper-104).  Alexandria,  VA. 

George,  A.,  Morris,  M.,  Galdorisi,  G.,  Raney,  C.,  Bowers,  A.,  &  Yetman,  C.  (2013,  June). 
Mission  composable  C3  in  DIL  information  environments  using  widgets  and  app  stores. 
In  Proceedings  of  the  18th  International  Command  and  Control  Research  and 
Technology  Symposium  (Paper-036).  Alexandria,  VA. 

George,  A.,  Morris,  M.,  &  O’Neil,  M.  (2014).  Pushing  a  big  rock  up  a  steep  hill:  Lessons 

learned  from  DoD  applications  storefront.  In  Proceedings  of  the  11th  Annual  Acquisition 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


- 181  - 


Research  Symposium  (Vol.  1;  pp.  306-317).  Monterey,  CA:  Naval  Postgraduate 
School. 

Guertin,  N.  H.,  Sweeney,  R.,  &  Schmidt,  D.  C.  (2015,  May).  How  the  Navy  can  use  open 
systems  architecture  to  revolutionize  capability  acquisition:  The  naval  OSA  strategy  can 
yield  multiple  benefits  (NPS-AM-1 5-004).  In  Proceedings  of  the  12th  Annual  Acquisition 
Research  Symposium.  Monterey,  CA:  Naval  Postgraduate  School. 

Kazman,  R.,  &  Carriere,  S.  J.  (1998).  Playing  detective:  Reconstructing  software 

architecture  from  available  evidence.  Journal  of  Automated  Software  Engineering,  6(2), 
107-138. 

Kendall,  F.  (2015,  April  9).  Implementation  directive  for  Better  Buying  Power  3.0 
[Memorandum], 

Meyers,  B.  C.,  &  Obendorf,  P.  (2001).  Managing  software  acquisition:  Open  systems  and 
COTS  products.  New  York,  NY:  Addison-Wesley. 

Reed,  H.,  Benito,  P.,  Collens,  J.,  &  Stein,  F.  (2012,  June).  Supporting  agile  C2  with  an  agile 
and  adaptive  IT  ecosystem.  In  17th  International  Command  and  Control  Research  and 
Technology  Symposium  (ICCRTS;  Paper-044).  Fairfax,  VA. 

Reed,  H.,  Nankervis,  J.,  Cochran,  J.,  Parekh,  R.,  &  Stein,  F.  (2014,  June).  Agile,  adaptive  IT 
ecosystem:  Results,  outlook,  and  recommendations.  In  Proceedings  of  the  19th 
International  Command  and  Control  Research  and  Technology  Symposium  (ICCRTS; 
Paper-01 1 ).  Arlington,  VA. 

Scacchi,  W.  (2002).  Understanding  the  requirements  for  developing  open  source  software 
systems,  IEE  Proceedings — Software,  149(1),  24-39. 

Scacchi,  W.  (2009).  Understanding  requirements  for  open  source  software.  In  K.  Lyytinen, 

P.  Loucopoulos,  J.  Mylopoulos,  &  W.  Robinson  (Eds.),  Design  requirements 
engineering:  A  ten-year  perspective  (LNBIP  14;  pp.  467-494).  Springer-Verlag. 

Scacchi,  W.  (2010).  The  future  of  research  in  free/open  source  software  development.  In 
Proceedings  of  the  ACM  Workshop  on  the  Future  of  Software  Engineering  Research 
(FoSER;  pp.  315-319),  Santa  Fe,  NM. 

Scacchi,  W.,  &  Alspaugh,  T.  (2012,  July).  Understanding  the  role  of  licenses  and  evolution  in 
open  architecture  software  ecosystems.  Journal  of  Systems  and  Software,  85(7),  1479- 
1494. 

Scacchi,  W.,  &  Alspaugh,  T.  (2013a,  May).  Streamlining  the  process  of  acquiring  secure 
open  architecture  software  systems.  In  Proceedings  of  the  10th  Annual  Acquisition 
Research  Symposium  (pp.  608-623).  Monterey,  CA:  Naval  Postgraduate  Schoool. 

Scacchi,  W.,  &  Alspaugh,  T.  (2013b,  May).  Processes  in  securing  open  architecture 

software  systems.  In  Proceedings  of  the  2013  International  Conference  Software  and 
System  Processes  (pp.  126-135).  San  Francisco,  CA. 

Scacchi,  W.,  &  Alspaugh,  T.  (2013c,  June).  Challenges  in  the  development  and  evolution  of 
secure  open  architecture  command  and  control  systems.  In  Proceedings  of  the  18th 
International  Command  and  Control  Research  and  Technology  Symposium  (Paper- 
098).  Alexandria,  VA. 

Scacchi,  W.,  &  Alspaugh,  T.A.  (2013d).  Advances  in  the  acquisition  of  secure  systems 
based  on  open  architectures.  In  Journal  of  Cybersecurity  &  Information  Systems,  1(2), 
2-16. 

Scacchi,  W.,  &  Alspaugh,  T.  (2015,  May).  Achieving  Better  Buying  Power  through 

acquisition  of  open  architecture  software  systems  for  web  and  mobile  devices  (NPS- 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-182- 


AM-15-005).  In  Proceedings  ofthe12th  Annual  Acquisition  Research  Symposium. 
Monterey,  CA:  Naval  Postgraduate  School. 

Software  architecture,  (n.d.).  In  Wikipedia.  Retrieved  from 
https://en.wikipedia.org/wiki/Software  architecture 

Velasco-Elizondo,  P.,  Dwivedi,  V.,  Garlan,  D.,  Schmerl,  B.,  &  Fernandes,  J.  M.  (2013). 

Resolving  data  mismatches  in  end-user  compositions.  End-user  development.  Springer 
Berlin  Heidelberg,  120-136. 

Womble,  B.,  Schmidt,  W.,  Arendt,  M.,  &  Fain,  T.  (2011).  Delivering  savings  with  open 
architecture  and  product  lines.  In  Proceedings  of  the  8th  Acquisition  Research 
Symposium  (pp.  8-13).  Monterey,  CA:  Naval  Postgraduate  School. 

Acknowledgments 

This  report  was  supported  by  grant  #N00244-16-1-004  from  the  Acquisition 
Research  Program  at  the  Naval  Postgraduate  School,  Monterey,  CA.  No  endorsement, 
review,  or  approval  implied.  This  paper  reflects  the  views  and  opinions  of  the  authors,  and 
not  necessarily  the  views  or  positions  of  any  other  persons,  group,  enterprise,  or 
government  agency. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


- 183- 


Architecting  Out  Software  Intellectual  Property  Lock-In:  A 
Method  to  Advance  the  Efficacy  of  BBP 


Maj  Chris  Berardi,  USAF — is  a  Chief  of  Staff  of  the  Air  Force  Captains  Prestigious  PhD  Fellow  at 
MIT.  Prior  to  beginning  his  PhD,  Maj  Berardi  served  as  both  an  acquisition  and  intelligence  officer  in 
the  USAF.  He  received  a  master’s  degree  in  Engineering  and  Management  from  MIT  and  a  Bachelor 
of  Science  in  Management  from  the  United  States  Air  Force  Academy,  [cberardi@mit.edu] 

Bruce  Cameron — is  a  Lecturer  in  Engineering  Systems  at  MIT  and  a  consultant  on  platform 
strategies.  His  research  interests  include  technology  strategy,  system  architecture,  and  the 
management  of  product  platforms.  Dr.  Cameron  received  his  undergraduate  degree  from  the 
University  of  Toronto,  a  master’s  degree  in  Technology  Policy,  and  a  PhD  in  Engineering  Systems 
from  MIT.  [bcameron@mit.edu] 

Dan  Sturtevant — is  Founder  and  CEO  of  Silverthread,  Inc.,  which  studies  complexity  in  large-scale 
software  systems;  develops  methods  for  quantifying  “technical  debt”;  and  creates  tools  that  help 
organizations  build  insight,  reduce  risk,  and  improve  business  outcomes  on  difficult  software  projects. 
Dr.  Sturtevant  has  an  undergraduate  degree  in  computer  engineering  from  Lehigh  University,  a 
master’s  degree  in  Engineering  and  Management  from  MIT,  and  a  PhD  in  Engineering  Systems,  also 
from  MIT.  [dan@silverthreadinc.com] 

Carliss  Baldwin — is  the  William  L.  White  Professor  of  Business  Administration  at  the  Harvard 
Business  School.  She  studies  the  process  of  design  and  the  impact  of  design  architecture  on  firm 
strategy,  platforms,  and  business  ecosystems.  Dr.  Baldwin  has  a  doctorate  and  MBA  from  Harvard 
Business  School,  and  an  SB  in  economics  from  MIT.  [cbaldwin@hbs.edu] 

Ed  Crawley — is  a  Professor  of  Aeronautics  and  Astronautics  and  Engineering  Systems  at  MIT. 
Professor  Crawley  received  a  ScD  in  Aerospace  Structures  from  MIT.  His  early  research  interests 
centered  on  structural  dynamics,  aeroelasticity,  and  the  development  of  actively  controlled  and 
intelligent  structures.  Recently,  Dr.  Crawley’s  research  has  focused  on  the  domain  of  the  architecture 
and  design  of  complex  systems,  [crawley@mit.edu] 

Abstract 

This  paper  works  to  understand  Department  of  Defense  (DoD)  contracting  trends  since  the 
beginning  of  Better  Buying  Power  (BBP).  By  using  data  publicly  available  from  the 
Governmentwide  Point  of  Entry  (GPE),  this  paper  concludes  that  there  are  no  clear  trends  in 
the  levels  of  competition  in  the  DoD,  as  measured  by  ratios  of  Justifications  and  Approvals 
(J&A)  to  contract  awards,  as  a  result  of  BBP.  However,  this  is  not  to  say  that  BBP  is 
ineffectual,  but  that  methodologies  are  still  needed  to  implement  the  guidance  outlined  in 
BBP.  To  that  end,  this  paper  proposes  a  methodology  to  identify  salient  data  rights  in 
computer  software.  Our  aim  is  to  provide  a  means  for  program  managers  to  understand 
which  data  rights  are  most  important  to  ensure  future  sustained  competition. 

Introduction 

Over  five  years  have  passed  since  the  first  version  of  Better  Buying  Power  (BBP) 
was  introduced  by  Secretary  of  Defense  Aston  Carter,  formerly  Under  Secretary  of  Defense 
for  Acquisition,  Technology,  and  Logistics  (USD[AT&L];  Carter,  2010).  In  that  time,  two 
updated  versions  of  BBP  were  released  by  the  current  USD(AT&L)  Frank  Kendall:  BBP  2.0 
(Kendall,  2013)  and  BBP  3.0  (Kendall,  2015).  Since  the  initial  BBP,  many  authors  offered 
critiques  of  the  BBP  strategies  (examples  include  Hasik,  2014;  Hill,  2013;  Huitink,  2014; 
Hunte  et  al.,  2015;  Layden  &  Arndt,  2012)  with  varying  sentiments  of  approval  or 
disapproval.  However,  within  these  critiques  are  two  reoccurring  patterns.  First,  the  majority 
of  critiques  are  made  using  qualitative  methods.  These  analyses  are  important,  but 
represent  a  somewhat  one-sided  story  without  any  accompanying  quantitative  analyses. 
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Second,  many  lament  that  BBP  is  akin  to  a  management  philosophy  that  lacks  any  insight 
into  implementation,  but  very  few  critics  offer  methods  to  aid  in  implementation.  One 
example  is  Hasik’s  (2014)  review,  “Carter  effectively  introduced  a  slew  of  new  rules  into  the 
Pentagon’s  bureaucracy,  but  he  and  his  successor  have  developed  few  mechanisms  for 
affecting  the  behavioral  change  beyond  issuing  a  memorandum”  (p.  17). 

Consequently,  our  work  herein  endeavors  to  serve  two  purposes.  First,  to  analyze 
data  from  the  Governmentwide  Point  of  Entry1  (GPE)  to  identify  any  trends  in  levels  of  DoD 
competition  since  the  introduction  of  BBP.  We  will  not,  however,  tread  into  complicated  tests 
of  statistical  significance  with  the  hope  of  identifying  minute  changes  in  competition.  Rather, 
our  goal  is  to  look  for  broad  changes  in  DoD  competition  patterns,  after  which  we  will  devote 
the  second  half  of  this  paper  to  explain  a  methodological  approach  to  aid  in  the  realization  of 
three  BBP  initiatives. 

State  of  Competition 

The  2015  Annual  Report  of  the  Performance  of  the  Defense  Acquisition  System, 
released  annually  by  the  office  of  the  USD(AT&L),  argues  that  competition  is  starting  to  rise. 
To  substantiate  this  statement,  the  report  uses  a  fractional  measure  of  contracts 
competitively  awarded  by  dollar  amount.  The  most  recent  measures  show  that  58.3%  of 
fiscal  year  (FY)  14  contracts,  by  dollar  amount,  were  competitively  awarded,  which  is  up 
from  57%  in  FY13  (USD[AT&L],  2015).  However,  this  methodology  is  sensitive  to  an  outlier 
bias,  where  a  few  large  contracts  awarded  competitively  (i.e.,  contracts  on  the  order  of 
magnitude  in  the  $100s  of  millions)  overshadow  the  many  smaller  contracts  awarded  using 
other  than  full  and  open  methods.  In  this  scenario,  there  may  be  competition  amongst  a 
small  number  of  large  contractors  on  big  contracts,  but  little  to  no  competition  amongst  the 
many  hundreds,  if  not  thousands,  of  contractors  on  the  small  contracts.  This  results  in 
metrics  that  reflect  a  large  quantity  of  competition  by  dollar  amount,  but  little  change  in  the 
actual  number  of  competitive  contracts  awarded.  Unfortunately,  no  methods  were  used  to 
control  for  this  bias  in  either  the  quantitative  analysis  or  the  interpretation  of  results.  We 
make  no  argument  that  the  data  presented  by  the  USD(AT&L)  is  overestimated  or 
underestimated,  only  that  more  analyses  are  needed  to  triangulate  the  actual  effects. 

Given  these  methodological  choices,  it  is  difficult  to  determine,  with  any  certainty, 
whether  competition  in  DoD  contracts  is  increasing  or  decreasing.  This  makes  determining 
the  efficacy  of  BBP  equally  difficult.  Consequently,  we  propose  an  independent  study  using 
data  from  the  GPE  to  triangulate  the  results  in  the  2015  Performance  of  the  Defense 
Acquisitions  System  report.  The  GPE  is  an  online  repository  for  U.S.  Government  business 
opportunities,  which  is  accessible  by  the  public.  This  is  an  ideal  source  of  data  because  DoD 
agencies  are  required  by  Federal  Acquisition  Regulation  (FAR)  5.201  to  post  synopses  of 
contracting  actions  above  $25, 000. 2  Furthermore,  the  contract  notices  on  the  GPE  cover  the 


1  “Governmentwide  point  of  entry  (GPE)  means  the  single  point  where  Government  business 
opportunities  greater  than  $25,000,  including  synopses  of  proposed  contract  actions,  solicitations, 
and  associated  information,  can  be  accessed  electronically  by  the  public.  The  GPE  is  located  at 
http://www.fedbizopps.gov”  (48  CFR  2.101 — Definitions). 

2  FAR  5.202(a)  outlines  14  exceptions  to  the  mandatory  contract  synopsis  policy  outlined  in  5.201 . 
The  exceptions  are  too  lengthy  to  enumerate  in  detail  herein;  however,  the  14  exceptions  are 
sufficiently  specific  to  ensure  the  maximum  amount  of  information  is  available  to  the  public. 
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spectrum  from  a  vehicle  maintenance  contract  at  an  Air  Force  base  in  South  Dakota  in  the 
tens  of  thousands  of  dollars  to  ACAT-type  programs  in  the  tens  of  millions  of  dollars.  From 
these  posted  contract  notices,  we  can  quantify  the  overall  DoD  procurement  activity  per 
fiscal  year. 

In  addition  to  contract  notices,  other  types  of  contracting  actions  are  also  posted.  Of 
particular  interest  are  Justifications  and  Approvals  (J&A),3  which  is  a  document  released  to 
the  public  when  the  DoD  uses  a  procurement  strategy  other  than  full  and  open  competition.4 
The  February  17,  2009,  revision  of  FAR  6.302  mandated  that  all  J&A  documents  are  posted 
to  the  public.5  Each  J&A  notice  on  the  GPE  is  an  artifact  which  reflects  a  lack  of  competition 
in  DoD  acquisition.  Consequently,  the  frequency  of  J&As  relative  the  overall  number  of 
contract  awards  provides  a  separate  indicator  of  competitiveness  within  the  DoD 
marketplace  that  is  not  sensitive  to  outlier  bias. 

Data  Collection 

Although  the  data  is  electronically  available  to  the  public,  it  is  made  available  through 
a  web  interface.  Consequently,  to  retrieve  the  data  a  web  scraper  was  built,  tested,  and 
employed  to  obtain  the  data  for  all  DoD-related  agencies.  These  agencies  and  their 
respective  number  of  contract  notices  on  the  GPE  are  outlined  in  Table  1 . 

Web  scraping  is  a  time-consuming  process  which  is  costly  to  both  the  provider  of 
information,  in  terms  of  increased  website  traffic,  and  the  scraper,  in  terms  of  bandwidth  and 
data  size.  In  an  effort  to  minimize  the  impact  of  scraping,  only  seven  variables  were 
collected  for  each  notice  posted  on  the  GPE:  name,  contract  number,  class  code,  agency, 
procurement  organization  (within  the  agency),  date  of  notice,  type  of  notice,  and  the  URL  for 
each  specific  notice  page. 


3  “Justification  and  Approval  (J&A)  is  a  document  required  to  justify  and  obtain  appropriate  level 
approvals  to  contract  without  providing  for  full  and  open  competition  as  required  by  the  Federal 
Acquisition  Regulation  (FAR)”  (Defense  Acquisition  University,  2015). 

4  For  those  unfamiliar  with  DoD  jargon,  “other  than  full  and  open  competition”  is  defined  as  any  sole 
source  or  limited  competition  contract  action  that  does  not  provide  an  opportunity  for  all  responsible 
sources  to  submit  proposals. 

5  Similar  to  the  FAR  5.202(a)  exception,  there  are  also  exceptions  for  Justifications  and  Approvals 
outlined  in  FAR  6.305(b)  and  (c). 
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Table  1 .  Number  of  Notices  Collected  by  Agency 


AGENCY 


NOTICES 


DEFENSE  LOGISTICS  AGENCY 
DEPARTMENT  OF  AIR  FORCE 
DEPARTMENT  OF  ARMY 


675,041 

219,025 

322,139 

274,984 

27,756 


DEPARTMENT  OF  NAVY 
OTHER  DEFENSE  AGENCIES6 


TOTAL  1,518,945 


In  addition  to  the  approximately  1.5  million  contract  notices  collected,  additional 
details  specific  to  each  J&A  were  also  scraped.  Similar  to  the  contract  notices,  the  scraping 
only  gathered  a  parsimonious  collection  of  variables  for  each  J&A:  name,  contract  number, 
contract  award  date,  FAR  authority,  service,  command,  program  management  office, 
classification  code,  and  North  American  Industry  Classification  System  (NAICS)  code.  The 
results  of  this  secondary  scrape  are  outlined  in  Table  2. 

To  control  for  threats  to  internal  validity  from  instrumentation  error,  the  total  number 
of  notices  was  cross  checked  against  the  number  displayed  on  the  GPE.  In  all  cases,  the 
number  of  notices  collected  by  the  web  scraper  match  the  number  of  entries  in  the  GPE 
archive.  This  ensures  no  portion  of  the  data  was  errantly  omitted  from  the  sample. 
Additionally,  to  control  for  lagging  policy  effects  caused  by  the  February  17,  2009,  revision  of 
the  FAR,  which  mandated  J&A  public  disclosure,  all  data  before  FY10  was  omitted.  The 
resulting  data  covers  FY10-FY15.  Additionally,  three  contract  notices  contained  dates  with 
years  greater  than  2016,  and  66  of  the  J&As  did  not  contain  a  FAR  authorization.  These 
data  were  excluded  from  the  final  sample  analyzed  below. 

Data  Analysis 

There  is  some  seasonality  in  both  the  contract  award  and  J&A  data,  which  is 
centered  around  the  end  of  each  fiscal  year,  see  Figure  1(a).  This  result  is  expected,  given 
the  race  to  obligate  funds  before  the  end  of  the  fiscal  year.  However,  what  is  unexpected  is 
the  difference  in  seasonality  between  contract  awards  and  J&As  in  both  FY12  and  FY13 
(see  Figure  1(a)).  This  phenomenon  is  most  likely  explained  by  the  Budget  Control  Act  of 
201 1 ,  commonly  referred  to  as  “sequestration,”  which  required  nine  annual  sequesters  of 
$109  billion.  The  first  of  the  annual  sequesters  took  effect  in  March  2013  (Van  Hollen,  2015), 
likely  stymying  the  number  of  contracts  awarded  both  before  and  after.  Interestingly,  there 
appears  to  be  no  sequestration  effect  on  the  number  of  J&As  during  the  same  time  periods. 
Additionally,  we  see  no  substantive  reduction  in  the  absolute  number  of  J&As  in  the  six 
fiscal  years  under  study. 

We  also  considered  the  fraction  of  J&As  relative  to  contracts  awarded  (see  Figure 
1(b)).  This  measure  shows  a  ratio  of  uncompetitive  contracts  to  competitive  contracts.  In  this 


6  Examples  of  Other  Defense  Agencies  include:  Defense  Advanced  Research  Projects  Agency 
(DARPA),  Uniformed  Services  University  of  the  Health  Sciences  (USUHS),  U.S.  Special  Operations 
Command  (SOCOM),  Department  of  Defense  Education  Activity  (DoDEA),  Defense  Commissary 
Agency  (DCA),  United  States  Transportation  Command  (TRANSCOM),  Defense  Finance  and 
Accounting  Service  (DFAS),  etc. 
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data  we  see  spikes  in  the  fractional  measure  at  the  end  of  FY1 1  and  during  FY13.  As 
previously  discussed,  the  FY13  numbers  are  most  likely  explained  by  sequestration. 
However,  the  spike  during  the  initial  quarter  of  FY12  appears  to  be  a  result  of  low  number  of 
contracts  awarded.  In  fact,  the  contract  awards  during  the  first  quarter  of  FY12  are  the 
lowest  out  of  any  quarter  between  FY10-FY15.  This  is  likely  explained  by  the  five 
successive  Continuing  Resolutions  appropriations  in  FY12.  Interestingly,  there  seems  to  be 
no  obvious  connection  between  the  introduction  of  BBP  initiatives7  and  the  fraction  of  J&As 
to  contract  awards.  If  we  look  at  the  six-month  exponentially  weighted  moving  average 
(EWMA)  of  the  fractional  measure,  we  see  that  even  after  three  iterations  of  BBP  DoD  levels 
of  competition  are  similar.  This  is  not  to  suggest  that  BBP  was  ineffectual  or  that  there  was 
no  impact  on  competition,  but  that  no  obvious  conclusion  can  be  made  based  on  the  data  in 
the  GPE. 

Lastly,  we  examined  a  moving  cross  correlation  between  contract  awards  and  J&As 
(see  Figure  1(c)).  The  data  have  a  static  correlation  of  0.55  ( p  =  5.9e  -  7).  Furthermore,  for 
the  large  majority  of  the  months  spanning  FY10-FY15,  there  is  a  positive  six-month  moving 
cross  correlation,  sometimes  as  high  as  0.96.  As  with  Figures  1(a)  and  1(b),  there  is  some 
variability  in  the  outcomes,  but  those  are  largely  explained  as  lagging  effects  from  the 
events  previously  discussed.  We  believe  this  to  be  an  important  measure  of  competition 
because  if  competition  rates  were  truly  improving,  we  would  expect  to  see  the  correlation 
between  contract  awards  and  J&A  move  closer  to  0,  or  no  effect.  While  the  data  from  the 
GPE  do  demonstrate  some  periods  of  no  correlation  between  J&A  and  contract  awards, 

94%  of  the  points  in  the  sample  have  a  positive  six-month  moving  cross  correlation. 


7  BBP  initiatives  are  dated  using  the  stamp  dates  from  the  USD(AT&L)  memorandums  which  directed 
implementation.  The  respective  dates  are  as  follows:  BBP  1 .0 — June  28,  2010;  BBP  2. — April  24, 
2013;  and  BBP  3.0— April  9,  2015. 
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Figure  1.  Contract  Awards  and  J&As 

Note.  All  subplots  in  Figure  1  are  drawn  using  a  calendar  year  x-axis,  not  a  U.S.  Government 
Fiscal  Year  x-axis.  (a)  Shows  the  frequency  of  contract  awards  (left  y-axis)  and  J&As  (right  y- 
axis)  as  a  time  series  sampled  on  a  monthly  frequency,  (b)  Shows  the  fraction  of  J&A  to 
contract  awards.  Callouts  on  (b)  denote  when  each  of  the  three  versions  of  BBP  were 
mandated  by  USD(AT&L).  To  remove  some  seasonality  from  the  fraction,  the  red  dashed  line 
shows  a  six-month  exponentially  weighted  moving  average  (EWMA).  (c)  Shows  a  six-month 
moving  cross  correlation  between  contract  awards  and  J&As. 

Discussion 

The  aforementioned  results  do  not  demonstrate  any  facts  with  statistical  certainty; 
however,  they  do  illustrate  that  the  data  are  noisy  and  influenced  by  multiple  exogenous 
events.  Additionally,  the  independent  analysis  of  GPE  data  do  not  necessarily  support  the 
clear-cut  conclusions  in  the  2015  edition  of  the  Performance  of  the  Defense  Acquisition 
System  report.  The  numbers  may  be  improving  by  proportion  of  dollars  competitively 
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awarded,  but  an  examination  of  the  frequency  of  uncompetitive  contracts  to  competitive 
contracts  shows  no  clear  improvement. 

Those  familiar  with  J&As  and  FAR  Part  6  will  rightly  point  out  that  not  all  J&As  are 
equal  in  their  ability  to  indicate  a  lack  of  competition.  This  is  true;  there  are  seven  different 
categories  of  J&As  (the  GPE  uses  eight  categories,  adding  a  subcategory  for  6.302-1). 
Table  2  enumerates  the  different  types  of  J&As  and  their  frequency  in  the  GPE  during  the 
period  FY10-FY15. 

Table  2.  GPE  Justifications  and  Authorization  Types  and  Respective  Frequency 


J&A  AUTHORIZATION 

COUNT 

FAR  6.302-1  -  Only  One  Responsible  Source  (Except  Brand  Name) 

13.922 

FAR  6.302-1  (C)  -  Brand  Name 

2.1 55 

FAR  6.302-2  -  Unusual  and  Compelling  Urgency 

1748 

FAR  6.302-3  -  Industrial  Mobilization;  Engineering.  Developmental  or  Research 

138 

Capability:  or  Expert  Services 

FAR  6.302-4  -  International  Agreement 

59 

FAR  6.302-5  -  Authorized  or  Required  by  Statute 

607 

FAR  6.302-6  -  National  Security 

5 

FAR  6.302-7  -  Public  interest 

671 

TOTAL 

19.305 

The  J&A  authorized  by  FAR  6.302-2  is  for  other  than  full  and  open  competition  in 
cases  of  “unusual  and  compelling  urgency.”  This  category  does  not  accurately  reflect  the 
DoD’s  inability  to  implement  a  competitive  process,  but  a  choice  to  exclude  competition  in 
the  interest  of  exigent  circumstances.  This  same  argument  could  be  made  for  all  J&A  types, 
except  FAR  6.302-1 .  However,  FAR  6.302-1  is  by  far  the  most  prevalent  type  of  J&A, 
accounting  for  approximately  83%  of  all  the  J&As  in  the  GPE  (see  Figure  2(a)).  Across 
FY10-FY15  we  see  that  the  average  rate  of  6.302-1  J&As  is  approximately  83%,  with  the 
percentage  trending  somewhat  higher  in  FY12  and  FY13.  Consequently,  it  is  safe  to  argue 
that  6.302-1  J&As  are  by  far  the  most  frequently  used  across  the  fiscal  years  examined  and 
that  by  removing  the  remaining  17%  from  the  sample  would  only  have  a  marginal  effect  on 
the  analyses  presented  in  Figure  1 ,  as  the  ratio  of  6.302-1  J&As  removed  would  be  nearly 
uniform  for  each  fiscal  year.  Interestingly  Figure  2(a)  does  not  show  any  trends  in  J&A  type, 
as  a  percentage  of  total  number,  across  six  fiscal  years.  Prior  to  this  analysis  we 
hypothesized  that  the  impact  of  BBP  (1. 0-3.0)  would  have  had  some  noticeable  impact  on 
the  quantity  and  type  of  J&As  used.  From  the  data  presented  in  Figure  1  and  Figure  2,  there 
are  no  easily  observable  results  that  would  support  this  hypothesis. 
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Figure  2.  J&As  Data 

Note.  All  subplots  in  this  figure  have  different  x  axes,  which  differs  from  Figure  1  which  used 
a  common  x  axis,  (a)  Depicts  the  percentage  of  each  J&A  type  by  fiscal  year.  The  dashed  red 
line  shows  the  average  number  of  6.302-1  and  6.302-1  (c)  J&As  across  all  fiscal  years,  (b) 
Depicts  the  total  number  of  each  J&A  type  by  service  from  FY10  through  FY15.  (c)  OLS 
regression  of  the  total  number  of  6.302-1  and  6.302-1  (c)  J&As  from  FY10  through  FY15. 
Although  FY15  showed  a  reduction  in  6.302-1  and  6.302-1  (c)  J&As,  the  overall  trend  is 
slightly  positive  with  a  coefficient  of  45  and  an  intercept  of  2,567.  The  shaded  area  represents 
a  confidence  interval  of  58,  which  corresponds  with  the  standard  error  of  the  estimate. 

The  nature  of  the  information  stored  in  the  GPE  allows  us  to  also  understand  which 
Service  is  generating  the  most  6.302-1  J&As  (see  Figure  2(b)).  Prior  to  undergoing  this 
analysis,  we  hypothesized  that  all  services  would  be  roughly  equal  in  the  number  of  J&As. 
However,  the  data  at  Figure  2(b)  illustrate  a  different  picture.  For  the  sample  period 


mtSTANTIA  PER  SCIENTIAL 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  191  - 


examined  the  Army  and  Navy  are,  by  a  large  margin,  the  biggest  producers  of  6.302-1  J&As 
(e.g.,  the  Army  generated  approximately  150%  more  6.302-1  J&As  than  the  Air  Force  in  the 
sample  period).  However,  it  is  important  to  remind  the  reader  that  these  numbers  do  not 
control  for  the  size,  in  terms  of  dollars,  of  each  J&A.  As  a  somewhat  hyperbolic  example, 
each  of  the  Air  Force’s  J&As  could  be  for  a  $1  billion  space  system  and  the  Army’s  J&As 
could  be  for  a  $1 ,000  rifle.  Therefore,  there  are  limitations  on  the  conclusions  which  can  be 
drawn  from  the  data  outlined  in  Figure  2(b).  Also  interesting,  the  Defense  Logistics  Agency 
seems  to  be  the  sole  user  of  the  FAR  6.302-7,  Public  Interest,  J&A,  accounting  for  97%  of 
all  the  6.302-7  J&As  issued  across  six  fiscal  years.  Given  the  universal  nature  of  the  FAR, 
we  did  not  expect  one  service  to  dominate  the  use  of  a  particular  type  of  J&A. 

The  J&A  data  thus  far  demonstrates  that  6.302-1  is  the  most  prevalent  type  as  a 
percentage  of  the  total  number  of  J&As,  but  is  there  a  trend  in  the  use  of  6.302-1  J&As 
across  the  fiscal  years  in  the  sample?  To  address  this  question,  we  fit  an  Ordinary  Least 
Squares  (OLS)  regression  to  the  number  of  6.302-1  J&As  across  the  six  fiscal  years  under 
examination  (see  Figure  2(c)).  Given  the  small  number  of  samples  the  OLS  regression 
model  would  not  be  useful  as  a  predictor  of  future  values  of  6.302-1  J&As;  however,  it  is  a 
rough  indicator  of  a  positive  trend  in  the  number  of  6.302-1  J&As.  If  the  levels  of 
competition,  measured  as  number  of  contracts  competed,  were  increasing,  we  would  expect 
a  negative  or  downward  trend  in  the  number  of  6.302-1  J&As  over  the  sample  period.  That 
being  said,  the  standard  error  for  the  OLS  estimator  is  58,  which  indicates  it  is  possible,  but 
perhaps  not  probable,  that  there  is  a  slightly  negative  trend  in  the  data. 

Given  that  we  now  know  6.302-1  is  the  most  prevalent  type  of  J&A  and  that  6.302-1 
use  is  likely  increasing,  it  is  worth  discussing  this  specific  FAR  authorization  in  more  detail. 
The  FAR  does  not  enumerate  all  possible  uses  of  6.302-1 ,  but  it  does  provide  guidance  on 
application  of  the  regulation.  In  doing  so,  it  provides  four  situations  in  which  the  authority  in 
6.302-1  may  be  appropriate.  It  is  important  to  note  that  this  list  is  not  intended  to  be  all 
inclusive.  These  four  situations  are  as  follows: 


(1)  When  there  is  a  reasonable  basis  to  conclude  that  the  agency’s  minimum 
needs  can  only  be  satisfied  by  — 

(i)  Unique  supplies  or  services  available  from  only  one  source  or  only 
one  supplier  with  unique  capabilities;  or, 

(ii)  For  DoD,  NASA,  and  the  Coast  Guard,  unique  supplies  or  services 
available  from  only  one  or  a  limited  number  of  sources  or  from  only 
one  or  a  limited  number  of  suppliers  with  unique  capabilities. 

(2)  The  existence  of  limited  rights  in  data,  patent  rights,  copyrights,  or  secret 
processes;  the  control  of  basic  raw  material;  or  similar  circumstances, 
make  the  supplies  and  services  available  from  only  one  source  (however, 
the  mere  existence  of  such  rights  or  circumstances  does  not  in  and  of 
itself  justify  the  use  of  these  authorities)  (see  Part  27). 

(3)  When  acquiring  utility  services  (see  41.101),  circumstances  may  dictate 
that  only  one  supplier  can  furnish  the  service  (see  41 .202);  or  when  the 
contemplated  contract  is  for  construction  of  a  part  of  a  utility  system  and 
the  utility  company  itself  is  the  only  source  available  to  work  on  the 


system. 


(4)  When  the  agency  head  has  determined  in  accordance  with  the  agency’s 
standardization  program  that  only  specified  makes  and  models  of 
technical  equipment  and  parts  will  satisfy  the  agency’s  needs  for 
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additional  units  or  replacement  items,  and  only  one  source  is  available. 

(FAR  6.302-1  (b)) 

These  four  situations  are  relatively  generic  with  the  exception  of  6.302-1  (b)(2)  and 
6.302-1  (b)(3),  the  former  of  which  governs  the  application  of  6.302-1  for  situations  where 
intellectual  property  issues  or  data  rights  limit  competition,  and  the  ladder  towards  a  narrow 
situation  where  utility  services  are  acquired.  These  guiding  situations  pose  an  interesting 
question:  Are  there  certain  categories  of  goods  or  services  for  which  6.302-1  is  used  more 
frequently  than  others  given  the  specificity  in  6.302-1  (b)(2)?  To  obtain  an  approximate 
measure  of  this  we  examine  the  procurement  classification  codes8  of  each  6.302-1  J&A. 
These  are  considered  an  approximate  measure,  as  the  contracting  official  has  final  authority 
on  which  procurement  classification  code  the  J&A  will  use;  therefore,  there  is  some  variance 
in  which  types  of  effort  fall  into  which  procurement  classification  code.  Consequently,  the 
GPE  encourages  interested  bidders  to  search  across  similar  classification  codes  (e.g.,  a 
bidder  interested  in  15-Aircraft  and  Airframe  Structural  Components  should  also  search  in 
16-Aircraft  Components  and  Accessories).  However,  unless  the  two  codes  are  similar  in 
domain  or  type  then  the  procurement  classification  codes  provide  a  good  guideline  (i.e.,  it 
would  be  difficult  to  argue  there  is  overlap  between  24-Tractors  and  14-Guided  Missiles). 
Table  3  outlines  the  top  10  Product  Service  Codes  (PSC)  and  Federal  Supply  Codes  (FSC) 
by  number  of  6.302-1  J&As. 


Table  3.  Top  10  Product  Service  Codes  (PSC)  and  Federal  Supply  Codes  (FSC) 

by  6.302-1  J&A 


Products 

Count  1 

1  Services 

Count 

16  --  Aircraft  components  &  accessories 

1,720 

R  --  Professional,  administrative,  and 

management  support  services 

981 

70  --  General  purpose  IT  equipment 

1,107 

Q  —  Medical  services 

920 

20  --  Ship  and  marine  equipment 

725 

D  --  Information  technology  services, 
including  telecommunications  services 

613 

58  --  Communication  detection,  & 
coherent  radiation  equipment 

666 

J  --  Maintenance,  repair  &  rebuilding  of 
equipment 

677 

65  --  Medical,  dental,  veterinary 
equipment  &  supplies 

611 

A  -  Research  &  Development 

339 

15  --  Aircraft  &  airframe  structural 
components 

560 

S  -  Utilities  and  housekeeping  services 

297 

59  -  Electrical  and  electronic  equipment 
components 

525 

U  --  Education  &  training  services 

197 

99  --  Miscellaneous 

506 

V  —  Transportation,  travel,  &  relocation 
services 

142 

66  --  Instruments  &  laboratory  equip 

463 

Z  -  Maintenance,  repair,  and  alteration  of 
real  property 

136 

28  --Engines,  turbines  &  components 

326 

L  --  Technical  representative  services 

100 

8  Procurement  classification  codes  are  truncated  versions  of  both  the  Federal  Supply  Codes  (FSC) 
and  Product  Services  Codes  (PSC).  For  example,  instead  of  using  the  four-digit  Federal  Supply  Code 
1620 -Aircraft  Landing  Gear  Components ,  the  procurement  classification  code  uses  only  the  first  two 
digits  16-Aircraft  Components  and  Accessories. 
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Table  3  shows  some  interesting  patterns  in  the  type  of  goods  and  services  procured 
using  6.302-1  J&As,  namely  the  largest  number  of  product  J&As  belongs  primarily  to  aircraft 
related  PSCs  (16,15,  and  28)  and  the  second  most  appears  to  be  electronic  equipment 
PSCs  (70,  58,  59,  and  66).  Viewing  these  results  under  a  Williamsonian  Transactional  Cost 
Economics  lens,  these  results  are  somewhat  intuitive  based  on  the  language  in  FAR  6.302- 
1(b)(2).  Products  with  a  high  asset  specificity  are  expected  to  carry  some  mode  of 
safeguard,  in  this  instance,  intellectual  property  protection  to  defend  against  transactional 
hazards  (Williamson,  1981).  Said  differently,  products  built  specifically  for  a  single  purpose 
with  little  opportunity  for  dual-use  in  the  commercial  market  (e.g.,  aircraft  landing  gear  on  a 
F-16  or  IT  equipment  designed  to  process  UAV  full  motion  video)  are  expected  to  carry 
intellectual  property  safeguards  to  protect  the  manufacturers’  ideas  and  investments.  Failure 
to  protect  such  intellectual  property  would  lead  to  a  situation  where  competitors  could  easily 
reproduce  the  original  manufacturer’s  goods,  a  transactional  hazard.  Therefore,  we  can 
expect  these  types  of  products  are  often  procured  under  a  6.302-1  J&A.  Although  not  a 
revelatory  conclusion,  this  loose  connection  between  intellectual  property  and  6.302-1  J&As 
adds  credence  to  many  of  the  intended  changes  suggested  in  BBP. 

Improving  Competition 

The  most  recent  iteration  of  Better  Buying  Power,  BBP  3.0,  outlines  three  strategies 
which  confront  the  issue  of  intellectual  property  in  DoD  procurement.  The  first,  Remove 
Barriers  to  Commercial  Technology  Utilization,  argues  that  the  DoD  should  capture  private 
sector  innovation  by  using  commercially  available  technologies  and  products,  but  directs 
further  analysis  of  the  implications  on  intellectual  property.  The  second  strategy,  Increase 
the  Productivity  of  Corporate  Independent  Research  and  Development  (IRAD),  targets  the 
misuse  of  IRAD  funds  by  defense  contractor  on  “de  minimis  investments  primarily  intended 
to  create  intellectual  property”  (Kendall,  2015)  to  secure  a  competitive  advantage  in  future 
DoD  contracts.  The  final  strategy,  Use  Modular  Open  Systems  Architecture  to  Stimulate 
Innovation,  argues  that  the  DoD  must  control  relevant  interfaces  to  ensure  competitors  with 
superior  products  are  not  occluded  from  competition  due  to  intellectual  property  restricted 
interfaces. 

The  commonality  across  these  three  strategies  is  the  management  of  intellectual 
property.  This  is  not  a  new  concept  in  DoD  procurement.  In  fact,  there  is  a  statutory 
requirement  for  the  DoD  to  manage  intellectual  property  in  the  John  Warner  National 
Defense  Authorization  Act  for  Fiscal  Year  2007,  which  “require[s]  program  managers  for 
major  weapon  systems  and  subsystems  of  major  weapon  systems  to  assess  the  long-term 
technical  data  needs  of  such  systems  and  subsystems  and  establish  corresponding 
acquisition  strategies  that  provide  for  technical  data  rights  needed  to  sustain  such  systems 
and  subsystems  over  their  life  cycle”  (Mazour,  2009,  citing  10  U.S.C.  §  2320(e)).  This  is  a 
difficult  statute  to  comply  with  because  it  asks  members  of  the  DoD  acquisition  community  to 
predict  what  data  rights  are  needed  in  the  future.  Being  that  there  is  no  readily  agreed  upon 
method  for  accomplishing  data  rights  forecasting,  Eli  Mazour  (2009),  in  his  article  for  Public 
Contract  Law  Journal,  correctly  points  out  that  the  easiest  way  to  comply  with  this  law  “is  to 
acquire  as  many  rights  as  possible  in  as  much  technical  data  possible”  (p.  681).  The  logic 
being,  if  one  acquires  all  possible  data  rights,  then  one  is  certainly  prepared  to  sustain  a 
weapon  system  over  its  life  cycle.  Mazour’s  comments  are  illustrative  of  the  problem  with 
both  statutory  requirements  and  with  the  BBP  strategies;  neither  offer  a  means  for  program 
managers  to  accomplish  what  is  required.  This  brings  us  to  a  major  question  confounding 
the  DoD  procurement  community — how  do  program  managers  determine  which  data  rights 
to  purchase? 
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In  the  second  half  of  this  paper  we  endeavor  to  provide  a  means  for  understanding 
intellectual  property  in  computer  software  and  determining  which  data  rights  should  be 
purchased  to  increase  the  likelihood  of  sustained  future  competition.  The  choice  of  software, 
versus  a  physical  system  (e.g.,  aircraft  components),  is  critical  for  three  reasons.  First,  our 
analyses  of  J&As  herein  identifies  electronic  equipment,  which  all  host  software,  as  an  area 
where  items  are  often  procured  using  other  than  full  and  open  competition.  Second,  the 
increased  DoD  emphasis  on  acquiring  open  source  software  and  open  architectures  is 
driving  the  acquisition  of  software  systems  with  a  complex  web  of  open  source  and  closed 
source  intellectual  property  regimes  (Carter,  2010;  Kendall,  2013,  2015).  Understanding  the 
interactions  between  these  vastly  different  intellectual  property  regimes  is  instrumental  for 
future  policy  decisions.  Third,  there  is  a  steady  increase  in  the  functions  performed  by 
software  in  major  DoD  weapon  systems.  Consider  the  juxtaposition  of  two  generations  of 
aircraft:  In  1970s  aircraft  (e.g.,  F-1 1 1 ),  around  20%  of  its  functions  were  performed  by 
software,  whereas  for  aircraft  in  the  early  2000s  (e.g.,  F-22),  80%  of  its  functions  are 
performed  by  software  (Ferguson,  2001).  The  role  of  software  and  the  intellectual  property 
of  software  are  becoming  increasingly  important  across  all  future  DoD  acquisitions. 

Intellectual  Property  Lock-In 

Before  discussing  our  methods  in  detail,  it  is  important  to  understand  the  intellectual 
property  mechanisms  that  prevent  competition.  Specifically,  intellectual  property  lock-in 
occurs  “when  switching  costs  outweigh  the  benefit  of  adopting  a  superior  new  product  [and] 
a  consumer  is  locked  in  to  her  incumbent  supplier”  (Breuhan,  1997,  p.  2).  This  switching 
cost  could  be  the  cost  of  a  new  product  itself,  the  redesign  of  a  system,  or  the  licensing 
costs  of  any  new  intellectual  property.  Described  in  a  narrative  manner,  lock-in  often  occurs 
when  companies  vie  for  a  share  in  a  new  market.  In  this  situation,  companies  compete  hard 
for  early  adopters  in  a  given  technology,  oftentimes  with  penetration  pricing  (Farrell  & 
Klemperer,  2007).  An  organization  becomes  locked-in  when  the  cost  of  switching  to  another 
technology  outweighs  the  benefit  of  adopting  a  superior,  sometimes  cheaper,  product. 

Taking  it  one  step  further,  the  concept  of  switching  costs  is  based  on  the 
substitutability  of  a  new  technology  or  component.  If  a  new  piece  of  technology  is  easily 
substitutable,  in  terms  of  time  and  money,  into  a  legacy  system,  then  we  argue  there  is  a 
relatively  low  switching  cost.  Conversely,  if  a  piece  of  technology  is  not  easily  substitutable, 
we  argue  that  with  this  lack  of  substitutability  comes  a  high  switching  cost  and  subsequently 
a  high  potential  for  lock-in.  What  defines  intellectual  property  lock-in,  as  opposed  to 
technological  lock-in,  are  switching  costs  determined  by  rights  to  intellectual  property 
(defined  in  the  DoD  lexicon  as  “data  rights”). 

Previous  work  on  the  evolution  of  software  design  and  modularity  provide  a  means  to 
assess  the  substitutability  of  files  in  software  with  little  a  priori  knowledge  on  the  functionality 
of  the  software  itself  (MacCormack,  Rusnak,  &  Baldwin,  2007).  This  work  builds  from 
previous  theory  on  the  architectural  design  (Baldwin  &  Clark,  2000)  and  research  on 
quantifying  modularity  in  software  (MacCormack,  Rusnak,  &  Baldwin,  2006).  Specifically, 
previous  work  studied  the  evolution  of  software  over  a  series  of  sequential  versions  to 
identify  the  most  important  factors  in  component  survival,  “which  is  an  indicator  of  the 
degree  to  which  components  can  be  removed  or  substituted”  (MacCormack  et  al.,  2007,  p. 

4)  and  shows  that  tightly-coupled  components  have  a  higher  probability  of  survival  as 
software  evolves,  making  them  “harder-to-kill.”  Since  this  measure  of  hardness-to-kill  is  a 
proximal  measure  for  substitutability,  it  should  also  serve  to  identify  those  components  which 
have  high  switching  costs  and,  ergo,  a  large  potential  for  lock-in. 
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Method 

Our  selected  method  applies  a  Design  Structure  Matrix  (DSM)  approach  to  analyze 
the  relationships  between  different  entities  in  a  software  system.  The  basic  approach  relies 
on  the  following  four  steps  (MacCormack  et  al.,  2006,  2007): 

1 .  Capture  a  network  representation  of  software  source-code  using  dependency 
extraction  tools. 

2.  Find  all  the  paths  (direct  and  indirect)  between  files  in  the  network  by 
computing  transitive  closure. 

3.  Calculate  visibility  scores  to  each  file,  which  represent  a  file’s  reachability 
from  other  files  or  ability  to  reach  other  files  in  the  network. 

4.  Organize  files  into  one  of  four  canonical  groups  based  on  visibility  scores. 

We  will  enumerate  the  basics  of  each  step  beginning  with  network  extraction.  There 
are  two  basic  choices  regarding  the  application  of  DSMs  to  software:  (1 )  the  level  of  unit 
analyzed,  and  (2)  the  type  of  dependency  between  units.  Regarding  the  unit  analyzed,  it  is 
possible  to  analyze  software  at  the  directory,  source  file,  and  function  levels.  In  this 
methodological  approach  the  source  file  is  used  as  the  unit  of  analysis,  which  is  also 
supported  by  prior  work  on  software  design  (Cataldo  et  al.,  2006;  Eick,  Graves,  &  Karr, 

2001;  Sturtevant,  2013).  There  are  also  choices  in  the  dependency  type  between  these 
source  files.  Keeping  with  previous  literature  (MacCormack  et  al.,  2006;  Rusovan,  Lawford, 

&  Parnas,  2007)  we  use  the  function  call.  A  function  call  is  an  instruction  in  the  software 
code  that  requests  a  specific  task  be  executed,  sometimes  making  the  request  internally  or 
of  another  source  file.  When  one  source  file  executes  a  function  call  that  requests  a  task  to 
be  completed  by  another  source  file,  we  characterize  this  as  a  directional  dependency 
between  the  two  source  files.  For  example,  if  a  function  call  in  file  X  calls  a  function  call  in 
file  Z,  we  would  say  that  file  X  depends  on  file  Z.  It  is  important  to  note  that  this  is  a 
directional  dependency  and  just  because  file  X  depends  on  file  Z,  it  does  not  imply  file  Z 
depends  on  file  Z.  This  first  order  dependency  extraction  is  accomplished  using  a 
commercial  call  extractor,  specifically  SciTools  Understand.9 

To  illustrate  the  output  of  step  1  and  the  process  of  step  2,  we  point  your  attention  to 
Figure  3.  In  this  toy  example  we  have  extracted  the  dependencies  from  a  piece  of  software 
that  contains  four  source  files  (i.e.,  D,  C,  A,  and  B).  Upon  completing  the  extraction,  we  see 
that  D  calls  C  and  C  calls  both  A  and  D.  This  is  a  direct,  or  level  1 ,  connection  between  the 
files.  The  resulting  DSM  is  an  illustration  of  the  same  connections  in  the  network  graph, 
except  arrayed  as  a  matrix.  The  DSM  is  read  from  right  to  left.  For  example,  starting  at  row 
C  from  left  to  right,  we  see  that  two  blue  dots  denote  file  C  is  connected  to  both  A  and  B. 

The  next  step  in  the  methodology  calls  for  identifying  all  the  direct  and  indirect  paths  through 
the  software.  This  full  set  of  paths  is  known  as  visibility  or  transitive  closure.  To  identify  the 
visibility  of  each  source  file  we  use  matrix  multiplication.10  Specifically,  by  raising  the  DSM  to 
successive  powers  of  n,  we  obtain  the  direct  and  indirect  dependences  that  exist  for  each 
successive  path  length  n.  Summing  these  n  matrices  yields  the  visibility  matrix,  which  shows 
both  direct  and  indirect  dependencies  between  source  files  for  all  possible  path  lengths  up 


9  See  www.scitools.com  for  more  details. 

10  Note  that  we  choose  to  include  the  matrix  for  n  =  0  when  deriving  the  visibility  matrix,  implying  that 
an  element  will  always  depend  on  itself. 
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to  the  maximum.  If  we  draw  our  attention  to  the  visibility  matrix  in  Figure  3,  we  now  see  two 
new  entries  for  source  file  D.  This  is  because  file  D  originally  connected  to  file  C,  but  file  C 
also  connected  to  files  A  and  B.  Therefore,  in  the  visibility  matrix  we  have  captured  this 
second  order  connection  between  D  to  A  and  D  to  B. 
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2  2  Simple  Network  (Transitive  Closure) 


Figure  3.  Example  Level  1  DSM,  Example  Transitive  Closure  Calculations,  and 

Example  Visibility  Matrix 

The  penultimate  step  is  to  calculate  the  visibility  scores  for  each  file  in  the  software. 
The  measures  of  visibility  are  derived  directly  from  the  visibility  matrix  (Figure  4  uses  the 
same  visibility  matrix  from  Figure  3).  Visibility  fan-out  (VFO)  is  calculated  by  summing  all  the 
dependencies  along  each  row  of  the  matrix,  including  the  diagonal.  For  example,  file  C  has 
a  VFO  of  3,  which  means  that  it  depends  on  75%  of  the  files  in  the  software  either  directly  or 
indirectly.  Visibility  fan-in  (VFI)  is  calculated  similarly,  instead  by  summing  down  the  columns 
of  the  visibility  matrix.  Continuing  the  example,  file  C  is  seen  by  both  itself  and  file  D. 


Fan-out  Visibility  (VFO)  -  Sum  along  rows  of  visibility 

matrix  and  divide  by  total  number  of  elements:  FOV 
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elements  that  depend  on  it  (or  call  functions  within  it) 

Figure  4.  Calculations  for  Fan-Out  Visibility  and  Fan-In  Visibility 

With  visibility  scores  calculated,  we  now  organize  each  file  into  one  of  four  types  of 
files  (see  Figure  5).  This  step  is  critical  because  previous  work  suggests  files  with  high  VFI 
and  high  VFO  are  statistically  significant  indicators  of  hardness-to-kill  (MacCormack  et  al., 
2007).  However,  high  VFO  by  itself  was  not  uniformly  significant  across  all  samples  in 
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previous  research,  suggesting  that  high  VFI  is  more  dominant  in  explaining  survival. 
Intuitively  this  makes  sense;  files  with  a  high  VFI  score  imply  they  are  relied  upon 
extensively  by  other  files  in  the  software.  Substituting  a  file  which  is  relied  upon  extensively 
by  other  files  is  difficult,  whereas  substituting  a  file  which  relies  upon  others  (i.e.,  high  VFO) 
is  relatively  easier.  Consequently,  we  conclude  the  two  types  of  components  which  are  both 
more  survivable  and  least  substitutable  fall  into  the  “shared”  and  “core”  classes  (highlighted 
in  red  on  Figure  5). 


Four  Canonical  Types  Of  Components 
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Indirect  Ffln-In 

Components: 

Figure  5.  Four  Canonical  Types  of  Files 

(adapted  from  Lagerstrom  et  al.,  2013) 

Case  Study 

To  illustrate  the  utility  of  this  methodology,  we  will  examine  a  piece  of  DoD 
developed  software  and  gain  an  understanding  of  which  files  have  high  switching  costs  and 
could  result  in  intellectual  property  lock-in.  Specifically,  we  will  examine  a  flight  simulator 
software  currently  under  sustainment  at  Air  Force  Materiel  Command.  The  end  goal  of  this 
case  study  is  to  show  how  the  method  above  can  be  utilized  to  identify  a  list  of  files  for 
which  the  data  rights  should  be  secured  to  ensure  future  sustained  competition. 

This  particular  piece  of  software  is  comprised  of  6,362  files  primarily  written  in  C++ 
and  Java.  Using  the  methodology  described  above,  calls  were  extracted,  visibility  metrics 
calculated,  and  each  file  was  organized  into  one  of  the  four  groups  in  Figure  5.  The  results 
are  depicted  in  DSM  at  Figure  6.  Recall  from  Figure  5  that  the  most  important  groups  to 
determine  substitutability  and  hardness-to-kill  were  the  “shared”  and  “core”  groups.  These 
two  groups  are  organized  in  the  upper  left  corner  of  Figure  6  and  are  labeled  accordingly. 
The  files  that  fall  into  either  shared  or  core  comprise  approximately  18%  software,  as  a 
percentage  of  total  files. 
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FliphI  SimulMar  Fifes  Sorted  by  VFI  ard  VFO 


Ho  iW.;nKi>tl% hmdr>4Mi 


Figure  6.  Flight  Simulator  Files  Sorted  by  Visibility  Fan-In  and  Visibility  Fan-Out 

We  argue  that  this  visual  output  and  categorization  of  files  should  assist  program 
managers  in  determining  which  data  rights  to  purchase.  Without  this  method  the  program 
manager  would  have  had  to  decide  which  data  rights  in  the  6,362  files  were  instrumental  for 
future  competition.  Further  exasperating  the  problem,  there  is  a  low  likelihood  that  the 
program  manager  has  any  formal  training  in  computer  science  or  software  development. 

The  attractiveness  of  our  methodology  is  that  it  allows  for  the  prioritization  of  data  rights 
without  any  understanding  of  the  actual  lines  of  code  in  each  file.  The  program  manager 
does  not  need  to  understand  the  function  calls  in  any  given  file  or  even  the  purpose  of  each; 
he  or  she  must  only  understand  that  one  file  calls  another.  From  this  we  can  identify  the  files 
which  are  difficult  to  separate  from  the  software  at  large  and  are  likely  to  survive  in  the 
software  throughout  multiple  versions.  In  acquiring  the  rights  to  just  this  small  percentage  of 
files,  we  argue  that  it  increases  the  likelihood  of  future  sustained  competition  because  the 
DoD  has  rights  to  the  subset  of  files  which  are  hardest  to  operate  the  software  without. 

Future  Work/Conclusion 

We  have  shown  that  there  are  no  clear  trends  in  the  levels  of  competition  in  the  DoD, 
as  measured  by  ratios  of  J&As  to  contract  awards,  as  a  result  of  BBP.  However,  this  is  not 
to  say  that  BBP  is  ineffectual,  but  that  methodologies  are  still  needed  to  implement  the 
guidance  outlined  in  BBP.  To  that  end,  we  proposed  a  methodology  to  identify  salient  data 
rights  in  computer  software,  thus  providing  a  means  for  program  managers  to  understand 
which  data  rights  are  most  important  to  ensure  future  sustained  competition. 


PBAlSTANTlA  PER  SCIENTIAL 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-  199- 


That  being  said,  there  are  limitations  to  our  proposed  method.  First,  we  make  no 
attempt  to  account  for  the  effects  of  file  specific  licenses  on  the  prioritization  of  data  rights. 
In  the  files  identified  in  the  flight  simulator  example,  there  could  be  open  source  files  which 
carry  varying  license  types11  (e.g.,  General  Public  License,  Lesser  General  Public  License, 
Creative  Commons,  BSD,  MIT,  etc.).  The  presence  of  one  or  more  of  these  licenses  in  the 
software  could  change  which  files  the  government  is  entitled  data  rights.  Further  work  is 
needed  to  combine  the  license  information  at  the  file  level  into  the  methodology  discussed 
herein.  However,  with  more  research  we  argue  this  obstacle  is  solvable. 
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Abstract 

Private  industry  and  the  Military  both  recognize  the  need  to  develop  mobile  applications 
(apps)  to  meet  the  growing  demand  for  delivering  content  in  a  way  that  supports  end-users’ 
needs  and  preferences.  The  U.S.  Navy  has  been  examining  all  the  conceivable  strategic, 
policy,  and  security  issues  surrounding  mobile  application  development  and  deployment,  but 
limited  Navy  commands  have  had  success  implementing  a  policy  and  development 
methodology  for  meeting  widespread  end-user  needs. 

One  exception  has  been  the  Program  Executive  Office  (PEO)  for  Enterprise  Information 
Systems  (EIS),  a  U.S.  Navy  Program  Executive  Office  whose  mission  is  developing  and 
sustaining  business  Information  Technology  (IT)  systems  for  the  Navy.  One  of  their  primary 
customers,  the  Chief  of  Naval  Personnel,  challenged  PEO  EIS  to  develop  a  strategy  and 
development  methodology  for  quickly  developing  mobile  applications  to  meet  a  variety  of 
Navy  Human  Resource  (HR)  needs. 

PEO  EIS,  through  a  designation  to  one  of  its  Program  Management  Offices  (PMOs) — PMW 
240,  or  the  “Sea  Warrior”  Program — employed  an  innovative  approach  for  design, 
development,  and  acquisition  of  mobile  applications  that  has  allowed  it  to  field  multiple  mobile 
applications  in  just  8-12  weeks  per  application  given  strong  customer  engagement.  To  date, 
PMW  240  has  fielded  eight  applications  in  the  past  year  with  dozens  more  in  the  planning 
and  development  phases. 

This  paper  will  share  the  innovative  methodology,  Systems  Engineering  Technical  Review 
(SETR)  process,  and  Federal  Acquisition  Regulation  (FAR)  insights  that  have  allowed  PMW 
240  to  field  mobile  apps  rapidly.  It  will  also  discuss  some  of  the  challenges  and  next  steps  to 
expanding  the  Navy  HR  mobile  application  capabilities.  Since  PMW  240  is  an  acquisition 
executor,  all  processes,  innovations,  and  insights  will  be  presented  from  a  practitioner 
perspective  in  hopes  of  benefiting  other  practitioner  organizations  that  require  mobile 
application  deployment  for  their  end-users. 

Background 

There  has  been  an  unprecedented  level  of  interest  across  the  U.S.  Navy  to  rapidly 
investigate  and  enhance  existing  mobile  technology  capabilities,  primarily  due  to  their 
familiarity,  convenience,  ease  of  use,  and  productivity  benefits.  This  investigation  is 
considering  implementations  that  leverage  Government  Furnished  Equipment  (GFE)  and 
“Bring  Your  Own  Device”  (BYOD)  models. 
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As  the  lead  organization  for  Navy  Enterprise  mobility,  the  Deputy  Chief  of  Naval 
Operations  for  Information  Warfare  held  a  Mobility  Summit  in  October  2014,  which  laid  the 
path  to  develop  a  holistic  view  of  Navy  enterprise  mobility  efforts — supporting  afloat,  ashore, 
and  forward  deployed  operating  environments.  As  a  result,  the  Enterprise  Mobility  Integrated 
Product  Team  (EMIPT)  stood  up  in  January  2015  to  serve  as  the  Navy’s  designated 
advisory  and  action  group  for  all  matters  pertaining  to  Navy  enterprise  mobility  efforts.  The 
team  defined  Enterprise  Mobility  as 

the  suite  of  technologies  and  solutions  that  provides  Navy  personnel  access 
to  information  any  time,  any  place,  and  from  any  device.  Access  may  be 
provided  via  government  and/or  commercial  infrastructure  utilizing  multiple 
device  capabilities,  and  related  network  and  applications  capabilities. 
(Department  of  the  Navy,  2012) 

Notwithstanding  the  demand  for  mobile  application  availability,  there  are  significant 
information  assurance  and  other  technical  and  policy  issues  being  actively  addressed 
across  the  Navy.  Leveraging  existing  guidance,  Navy  Manpower,  Personnel,  Training,  and 
Education  (MPT&E)  leadership  initiated  its  Mobile  Application  Management  effort  using  the 
support  of  the  Program  Executive  Office  for  Enterprise  Information  Systems  (PEO 
EIS)/PMW  240  Sea  Warrior  Program  to  develop  and  deploy  mobile  applications  for  the 
MPT&E  domain.  A  key  element  of  the  PMW  240  tasking  is  to  build  on  government  and 
commercial  best  practices,  document  its  business  and  technical  management  processes, 
and  lay  out  a  path  for  institutionalizing  MPT&E  mobile  application  management  practices. 
This  tasking  is  being  performed  by  the  MPT&E  Mobility  Team,  staffed  by  PMW  240. 

The  PMW  240  Mobility  Team  develops,  oversees  IA  accreditation,  tests,  deploys, 
and  supports  mobile  applications  based  on  requirements  from  the  MPT&E  user  community. 
The  PMW  240  Mobility  Project  allows  for  the  rapid  development  and  deployment  of  mobile 
applications  to  meet  both  end  user  and  Navy  leadership  demand  signals,  with  the  unique 
focus  of  providing  these  applications  on  BYOD  versus  GFE  platforms  and  devices. 

PMW  240’s  specific  involvement  in  mobile  application  development  began  when  the 
Chief  of  Naval  Personnel  (CNP)  challenged  PMW  240  to  build  two  Navy  lieutenants’  mobile 
application  concept  for  Division  Officers.  Six  months  later,  PMW  240  not  only  delivered  the 
eDIVO  (electronic  Division  Officers;  see  Figure  1)  application,  but  also  the  framework  for  all 
future  MPT&E  mobile  applications.  PMW  240  has  delivered  eight  more  information  and 
training  mobile  applications  since  eDIVO,  achieving  a  normal  time  to  deliver  from  concept 
approval  in  less  than  four  months,  with  the  development  pipeline  queue  filled  with  mobile 
applications  from  Sailors  and  functional  business  owners. 
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Figure  1.  eDIVO  Guide 


Problem  Statement 

PMW  240  was  tasked  with  quickly  developing  and  delivering  MPT&E  mobile 
applications  to  Sailors  on  their  personal  mobile  devices.  However,  unlike  most  commercial 
mobile  leaders  who  can  define  and  streamline  their  own  procurement  processes,  a  specific 
mobile  application  acquisition  process  did  not  exist  separately  from  the  standard  weapon 
system  acquisition  processes  that  PMW  240  could  leverage.  Using  the  standard  processes 
which  were  developed  for  large-scale  Department  of  Defense  (DoD)  weapon  systems  would 
have  returned  lengthy  development  schedules  and  increased  costs,  which  was 
unacceptable  to  the  Chief  of  Naval  Personnel  and  PEO  EIS. 

Innovative  Solutions  Approach 

To  address  the  challenges  articulated  by  the  problem  statement,  PMW  240 
recognized  it  had  to  acquire  mobile  applications  quickly  and  inexpensively.  To  achieve  this 
goal,  PMW  240  decided  to  use  a  robust  framework  provided  by  an  acquisition  process 
already  tailored  for  IT — the  Abbreviated  Acquisition  Programs  (AAP)  and  Non-Designated 
Program  process.  This  IT  acquisition  process  was  a  necessary  first  step,  but  PMW  240  also 
recognized  that  it  needed  to  “fine  tune”  and  adapt  that  existing  framework  into  one  that  could 
successfully  deliver  lightweight  and  secure  mobile  applications  to  their  end  users  within 
months  of  initial  conception  without  compromising  the  appropriate  quality  control  and 
security  checks  inherent  in  the  current  process.  Figure  2  illustrates  how  PMW  240 
innovatively  tailored  the  standard  weapon  system  acquisition  process  to  address  its  mobile 
application  delivery  challenge. 
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Figure  2.  Robust  DoD  IT  Acquisition  Lifecycle  Process  Tailored  to  the  PMW  240 

Mobile  Application  Process  and  Innovations 

PMW  240  has  now  successfully  implemented  the  acquisition  lifecycle  process 
outlined  in  Figure  2.  The  remainder  of  this  section  will  outline  specific  activities  of  the  Idea, 
Acquisition,  Development,  and  Sustainment  phases  of  this  process,  identify  core  principles 
upon  which  mobile  application  acquisition  is  being  executed,  and  discuss  specific 
acquisition-related  innovations  in  each  of  the  four  phases  of  the  Figure  2  lifecycle  process 
(Cochrane  &  Brown,  2010). 

Mobile  Acquisition  Lifecycle  Overview 

The  High-Level  Operational  Concept  graphic  in  Figure  3  depicts  the  streamlined 
process  the  PMW  240  Mobility  Team  uses  to  identify  mobile  application  requirements,  then 
progressively  lead  those  requirements  through  a  series  of  executable  systems  engineering 
and  project  management  phases  and  decisions  to  a  fully  functioning  and  sustainable  mobile 
application  (PMW  240  Sea  Warrior  Program,  2015b). 


PBAlSTANTlA  PER  SCIENTIAL 
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Figure  3.  PMW  240  Mobile  Application  Development  Process 


The  entire  mobile  application  development  process  flows  from  left  to  right  through 
the  Idea,  Acquisition,  Development,  and  Sustainment  phases,  with  specific  and  important 
activities  involving  both  functional  owners  (customers)  and  PMW  240  in  each  phase. 

The  process  starts  in  the  Idea  phase  with  the  generation  of  ideas  for  new  apps  being 
presented  to  and  evaluated  by  the  Mobile  Application  Group  (MAG),  a  governance  body  that 
prioritizes  mobile  application  development.  MAG  approved  applications  are  assigned  to  the 
PMW  240  Mobility  Team  for  acquisition  (PMW  240  Sea  Warrior  Program,  201 5e). 

The  Acquisition  Phase  starts  when  approved  ideas  are  more  formally  defined 
through  the  generation  of  acquisition  documents  and  data.  Through  PMW  240’s  streamlined 
acquisition  process  and  document  templates,  the  team  is  able  to  rapidly  move  on  these 
application  ideas.  The  PMW  240  team  then  identifies  an  acquisition  strategy,  works  with  the 
application  product  owner  (Navy  subject  matter  expert  organization  who  will  own  the  content 
of  the  application  after  development)  to  create  a  functional  requirements  document  (FRD), 
executes  a  Product  Owner  Agreement  (POA)  with  that  owner,  and  conducts  a  project  kick¬ 
off  with  application  product  owners  to  ensure  their  understanding  and  support  of  the 
application’s  readiness  for  development. 

After  project  kickoff,  PMW  240  enters  the  Development  phase  by  conducting  a  series 
of  tailored  systems  engineering  technical  review  (SETR)  events  that  guide  the  project 
through  requirements  refinement,  design,  development,  test,  and  production  readiness 
decisions  before  the  app  is  deployed  for  use.  Those  SETR  events  will  be  described  in  more 
detail  later  in  this  paper.  Applications  may  be  developed  internally  by  Navy  software 
developers,  by  PMW  240  contracted  software  developers,  or  externally  by  third  party 
developers  who  are  sponsored  by  Navy  MPT&E  functional  leads  or  independent  submitters. 
The  outputs  of  the  Acquisition/Development  Phase  are  a  fielded  mobile  application 
published  on  designated  application  stores. 

In  the  Sustainment  Phase,  the  Product  Owner  provides  updated  content  as  needed 
to  the  developer  who  updates  the  mobile  app  for  publication  via  the  app  store.  Both  the 
Product  Owner  and  the  PMW  240  Mobility  Team  monitor  feedback  on  content,  functionality, 
usability,  and  user  experience  to  determine  upgrades  or  retirement  for  the  app. 

Mobile  Application  Core  Principles 

In  addition  to  the  overarching  acquisition  lifecycle  and  development  process  it 
developed  and  is  following,  PMW  240  recognized  that  it  needed  to  identify  and  follow  some 
core  principles  to  also  guide  its  mobile  application  acquisition  efforts.  These  core  principles 
provide  a  solid  strategic  foundation  on  which  PMW  240  bases  its  mobile  application 
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acquisition  and  ensure  that  certain  performance  and  compliance  requirements  are  met 
(PMW  240  Sea  Warrior  Program,  2015a).  These  principles  are 


•  Simplicity  and  Rapid  Deployment — Mobile  application  projects  should  be 
designed  with  rapid  deployment  and  simplicity  in  mind,  wherever  possible. 

•  Performance — The  application  must  follow  standard  iOS  and  Android 
development  practices  to  ensure  a  normal  level  of  memory  consumption. 

•  Security — The  application  must  adhere  to  requirements  and  specifications 
outlined  in  the  Cybersecurity  Mobile  Application  Checklist,  Fortify  scans,  and 
their  respective  references. 

•  Compliance — The  application  must  comply  with  standard  mobile  platform 
vendor  development  guidelines  outlined  in  official  licensing  and  distribution 
agreements. 

•  User  Documentation  and  Training — The  application  should  make  all 
documentation,  lifecycle  management,  and  training  information  publically 
available  as  needed.  Application  tutorials  are  a  preferred  method  for  training 
users  of  mobile  applications  how  to  perform  necessary  functions  to  utilize  the 
application  effectively. 

•  Maintenance/Sustainment — Mobile  application  projects  should  be  designed 
to  reduce  the  burden  of  maintenance  and  other  sustainment  actions.  Before 
using  any  feature  or  supporting  software,  a  developer  must  first  search  for 
reports  indicating  software  and  support  will  not  sunset  in  the  near  future. 

Also,  the  developer  must  compare  alternatives  with  respect  to  proven 
software  and  support  longevity,  and  reputation  for  ease  of  maintenance. 

•  Feedback — The  user  must  have  the  capability  to  email  feedback  directly  to 
the  NAVY  31 1  helpdesk.  In  addition,  mobile  application  projects  will  use  a 
Commercial  Off  the  Shelf  (COTS)  capability  to  capture  feedback  within  the 
application  and  subsequently  collate,  tag,  and  send  that  data  to  NAVY  31 1 
when  appropriate. 

o  Each  application  will  be  issued  its  own  email  address  to  facilitate 
communication  between  the  COTS  software  and  NAVY  31 1 . 

o  The  COTS  software  also  provides  the  capability  to  capture  feedback 
from  various  App  Stores,  Facebook,  Twitter,  and  other  social  media 
resources,  creating  tickets  from  the  feedback  it  discovers. 


Idea  Phase  Innovations 

Although  the  Idea  phase  of  the  lifecycle  is  relatively  short  and  simple,  PMW  240  has 
applied  innovative  guidance  and  tools  to  both  accelerate  and  simplify  this  phase. 

Streamlined  Technical  and  Programmatic  Documentation 

Agile  Mobility  Plan 

PMW  240  developed  the  Agile  Mobility  Plan  (AMP)  to  provide  technical  information 
related  to  the  development,  cybersecurity,  testing,  deployment,  and  sustainment  of  PMW 
240  mobile  applications.  The  information  contained  in  the  document  represents  what  is 
common  to  all  PMW  240  mobile  application  investments.  The  AMP  is  the  blueprint  for  the 
technical  conduct  and  control  of  PMW  240  mobile  applications  from  inception  through 
sustainment.  As  a  lightweight,  tailored  version  of  the  PMW  240  Systems  Engineering  Plan 
(SEP),  this  25-page  document  (innovatively  short  and  concise)  highlights  only  the  aspects  of 
systems  engineering  that  are  prevalent  to  the  mobile  application  lifecycle.  This  allows  for  a 
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purposeful,  pointed  document  resulting  in  the  rapid  development  and  deployment  of  mobile 
applications  to  meet  both  end  user  and  Navy  leadership  demand  signals  (PMW  240  Sea 
Warrior  Program,  2015b). 

Agile  Mobility  Management  Plan 

PMW  240  also  developed  the  Agile  Mobility  Management  Plan  as  a  companion 
document  to  the  AMP.  The  Agile  Mobility  Management  Plan  provides  for  management  and 
governance  of  PMW  240  mobile  application  investments.  The  document  specifies  and 
delegates  the  Mobility  Project  Decision  Authority  (MDA)  from  the  PMW  240  Program 
Manager  down  to  a  lower  level,  the  Principal  Assistant  Program  Manager  (PAPM),  for 
expedited  decisions  and  more  availability.  Through  delegation  of  this  authority  within  the 
program  office,  project  milestones,  management  policies,  and  governance  decisions  occur 
at  a  more  rapid  pace,  permitting  the  PMW  240  team  to  deliver  more  applications  in  less  time 
(PMW  240  Sea  Warrior  Program,  2015a). 

Mobile  Adjudication  Board 

PMW  240  established  the  Mobility  Adjudication  Board  (MAB)  to  manage 
requirements,  defect  resolution,  and  other  mobile  application  project  issues  as  required;  it  is 
also  a  forum  used  to  implement  the  fundamental  change  management  process  of 
configuration  control  during  planning,  development,  deployment,  and  sustainment.  The 
Mobility  Adjudication  Board  Charter  (MABC)  enables  a  disciplined  approach  and  visibility  for 
the  approval,  disapproval,  and  prioritization  of  new  or  existing  requirements.  It  is  a  critical 
component  to  maintaining  the  known  configuration  and  ensuring  all  changes  are  approved 
prior  to  implementation. 

The  MABC  has  the  Scope  of  Authority  (SoA)  for  mobile  app  changes  to  the 
configuration  baseline  during  the  planning,  development,  deployment,  and  sustainment 
phases  of  the  program.  Mobility  projects  normally  progress  at  a  higher  rate  of  speed  than 
standard  web  application  endeavors.  For  this  reason,  a  single,  very  lightweight  governing 
structure  handles  configuration  management  oversight  during  development  (normally,  more 
cumbersome  Program  Review  Board  [PRB]  or  Configuration  Control  Board  [CCB]  structures 
in  full  IT  system  acquisitions).  This  innovatively  lightweight  structure  ensures  issues  are 
handled  in  a  timely  manner  by  the  appropriate  oversight  and  combines  two  traditional 
processes — PRB  and  CCB — into  a  single  efficient  review  team  (PMW  240  Sea  Warrior 
Program,  201 5d). 

Multiple  Entry  Points 

PMW  240  receives  mobile  applications  that  fall  under  various  states  of  maturity 
within  the  lifecycle  and  allows  for  all  to  enter  into  its  lifecycle  process.  Most  applications  are 
proposed  in  the  form  of  less  mature  ideas,  but  some  are  maturing  and  already  in  some 
phase  of  a  development  state,  or  are  fully  built  and  ready  to  be  published  into  the  application 
store.  To  conserve  resources  and  recognize  the  lifecycle  maturity  of  these  various 
applications,  PMW  240  considers  the  current  state  of  the  application  to  determine  where 
and  how  to  categorize  it.  This  allows  for  a  customized  yet  expedited  entry  into  the 
application  store  while  verifying  the  application  meets  the  proper  exit  criteria  for  deployment. 

Idea  Mailbox 

PMW  240  acquires  application  ideas  through  a  variety  of  sources,  including 
leadership  direction,  command  interest,  and  line  of  business  owner  ideas.  PMW  240  also 
utilizes  a  digital  mailbox  advertised  on  Navy  media.  This  innovative  mailbox,  seen  in  Figure 
4,  navyapps@navy.mil,  receives  ideas  for  new  applications  from  both  civilians  and  Sailors 
and  is  checked  on  a  weekly  basis  to  ensure  new  and  fresh  ideas  for  applications  from 
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practitioners  are  in  the  forefront  of  application  development  consideration.  This  mailbox  is  an 
innovative  approach  to  soliciting  mobile  application  ideas  directly  from  the  end-users  who 
will  benefit  from  them. 


Solve  a  problem,  save  time  or 
improve  a  process 


NavyApps@navy.mil 


We  might  just  make  an  app  for  that! 


Figure  4.  PMW  240’s  Application  Mailbox 


Mobile  Action  Group  Review  and  Approval 

On  a  quarterly  basis,  PMW  240  briefs  the  collected  application  ideas  to  the  Mobile 
Application  Group  (MAG)  for  investment  approval.  If  an  application  is  not  chosen  for 
immediate  investment,  an  informational  quad  chart  detailing  salient  information  for  each 
application  idea  is  entered  into  the  Mobility  Team’s  application  backlog  and  will  be 
reconsidered  by  the  MAG  at  the  following  quarterly  meeting.  If  the  MAG  approves  an 
application  idea,  the  PMW  240  Mobility  Team  performs  required  contracting  actions  which 
signal  the  beginning  of  the  acquisition  phase.  Using  a  quarterly  time-driven  review  period 
keeps  the  investments  current  and  provides  PMW  240  with  regular  direction  on  applications 
to  best  align  with  the  leadership  and  end-user  interest.  Such  frequent  review  of 
requirements  and  prioritization  is  innovative  for  IT  acquisition,  to  say  nothing  of  standard 
DoD  5000  weapon  systems  prescribed  processes  (PMW  240  Sea  Warrior  Program,  201 5e). 

Acquisition  Phase  Innovations 

The  acquisition  phase  of  anything  the  DoD  procures  is  not  generally  considered  to 
be  a  space  where  innovation  can  flourish.  However,  PMW  240  has  implemented  innovative 
approaches  to  allow  agility  in  acquiring  new  mobile  applications. 

FAR/Contracting 

Per  the  Federal  Acquisition  Regulation  (FAR),  contract  negotiation  and  execution 
can  require  extensive  effort  and  wait  time  for  an  initial  contract  award  and  additional  time  for 
follow-on  task  order  awards  against  an  Indefinite  Delivery/Indefinite  Quantity  (ID/IQ)  or 
Multiple  Award  Contract  (MAC)  vehicle.  To  support  rapid  mobile  application  development, 
PMW  240  quickly  recognized  it  needed  a  very  flexible  and  responsive  contracting  strategy 
and  associated  vehicle  that  FAR-prescribed  competitive  or  negotiated  contracting 
procedures  might  not  allow. 

As  a  result  of  the  unique  MNP  mobile  app  development  needs,  PMW  240  conducted 
market  research  to  identify  industry  standard  timelines  and  costs  for  developing  the  types  of 
mobile  apps  under  consideration.  Based  on  that  data,  PMW  240  alpha-negotiated  an  ID/IQ 
contract  vehicle  with  an  economically-disadvantaged  woman-owned  small  business.  That 
vehicle  contained  Firm-Fixed  Price  (FFP)  Contract  Line  Item  Numbers  (CLINs)  for  small, 
medium,  and  large  application  development,  as  well  as  for  maintenance  tasking  on  an  app 
by  app  basis.  The  CLIN  values  were  based  on  the  industry  standard  costs  and  allow  PMW 
240  to  award  new  task  orders  (TOs)  selecting  the  needed  CLINs  in  a  matter  of  days.  That 
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TO  award  speed  significantly  decreases  the  overall  acquisition  phase  time  requirements 
(PMW  240  Sea  Warrior  Program,  2015a). 

Product  Owner  Agreement 

Prior  to  beginning  any  development,  the  Mobility  Team  works  with  the  Product 
Owners  to  negotiate  the  Product  Owner  Agreement  (POA).  The  POA  is  a  lightweight 
document  comparable  to  a  larger  program’s  Memorandum  of  Agreement  (MOA)  that 
explains  the  responsibilities  of  the  Product  Owner  and  PMW  240  throughout  the  application 
lifecycle  and  ensures  that  an  application  is  maintained  following  publication  into  the 
application  stores.  The  POA  explains  that  the  Product  Owner  is  responsible  for  any  content- 
related  changes,  including  notifying  the  PMW  240  team  of  any  policy,  link,  or  material 
updates,  and  the  PMW  240  team  is  required  to  handle  any  technical  changes,  such  as  bug 
fixes  and  operating  system  updates.  This  document  is  negotiated  and  signed  by  the  two 
participating  teams  and  reposed  under  configuration  control.  The  lightweight  nature  of  the 
POA  is  innovative  in  that  it  allows  signature  at  a  lower  organizational  level,  so  it  requires 
less  oversight,  allowing  the  development  on  the  application  to  begin  sooner  (PMW  240  Sea 
Warrior  Program,  2016a). 

FRD  Templates/Flexible  Requirements  Gathering 

In  addition  to  the  POA,  PMW  240  requires  a  Functional  Requirements  Document 
(FRD)  to  be  completed  and  signed  before  beginning  development.  The  Mobility  Team  has 
three  FRD  templates  depending  on  the  type  of  the  application  to  be  built:  an  aggregated 
content  application,  a  training  application,  and  a  hybrid  application  (content  and  training). 
The  most  fitting  template  is  then  customized  through  a  series  of  rapid  meetings  with  the 
PMW  240  team  and  the  Product  Owners  and  reviewed  by  the  development  team  for  any 
needed  clarification.  Once  all  parties  are  confident  that  the  FRD  captures  the  vision  for  the 
application,  the  document  is  signed  out  and  the  development  phase  can  begin  with  the 
Tailored  Mobile  Design  Review  (TMDR).  This  innovatively  rapid  requirements  gathering  and 
clarification  process  using  these  pre-defined  templates  generally  requires  no  more  than  10 
business  days,  which  is  an  extremely  short  timeline  compared  to  standard  IT  and  weapon 
systems  acquisitions  timelines  for  similar  activities  (PMW  240  Sea  Warrior  Program,  2015c). 

Development  Phase  Innovations 

The  Development  phase  of  the  lifecycle  is  usually  the  longest  phase  of  any 
acquisition,  and  it  is  for  PMW  240  mobile  applications  as  well.  To  decrease  required 
development  time  as  much  as  possible,  PMW  240  has  implemented  innovative  guidance 
and  oversight. 

“Official”  Developer  Account 

With  a  diverse  set  of  product  owners  working  with  PMW  240  to  develop  the 
applications,  it  is  important  to  adhere  to  specific  standards  and  meet  certain  thresholds 
when  it  is  time  for  production.  PMW  240  established  developer  accounts  for  the  public 
application  stores  (Apple  and  Google)  and  is  the  clearing  house  for  all  official  Navy  MPT&E 
mobile  applications.  The  streamlined  process  for  publication  and  production  ensures  each 
application  meets  the  exit  criteria  for  the  PMW  240  process  and  the  entrance  criteria  for 
these  public  application  stores.  The  efficiency  of  this  singular  clearing  house  point  provides 
a  structured  process  and  effective  release  methodology. 

Tailored  Technical  Events 

The  PMW  240  Technical  Event  Process  (TEP)  guidebook  provides  guidance  for 
planning  Systems  Engineering  Technical  Reviews  (SETR)  events.  However,  that  document 
generally  guides  development  through  a  waterfall  approach  that  requires  months  and  even 
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years  of  technical  events  to  deliver  working  software.  PMW  240  innovatively  tailored  the 
SETR  guidance  to  match  the  agile  methodology  it  has  implemented  to  deliver  MPT&E 
mobile  apps  in  weeks  and  months  instead  of  years.  The  following  are  the  critical  tailored 
technical  events  required  to  track  progress  for  each  mobile  application.  Each  of  the  four 
technical  events  listed  below  are  one  hour  in  length  and  require  participation  from  the 
Product  Owners,  PMW  240,  the  development  team,  and  representatives  from  Cybersecurity, 
the  Public  Affairs  Office  (PAO),  Enterprise  Change  Management  (ECM),  Configuration 
Management  (CM),  Test,  and  Logistics. 

•  Kickoff  Meeting  (KO) — After  a  mobile  app  project  receives  MAG  approval,  a 
Kickoff  meeting  is  held  to  introduce  team  members  from  different 
competencies  and  stakeholders  to  establish  the  expectations  for 
development/deliver,  general  procedures  to  be  followed,  priorities,  schedule, 
and  clear  assignment  of  roles  and  responsibilities. 

•  Tailored  Mobile  Design  Review  (TMDR) — The  TMDR  is  a  tailored 
combination  of  three  standard  technical  reviews:  System  Requirements 
Review  (SRR),  System  Functional  Review  (SFR),  and  Preliminary  Design 
Review  (PDR).  Conducted  by  the  Mobility  Assistant  Project  Manager- 
Engineer  (APM-E),  this  review  ensures  the  preliminary  design  of  the 
application  meets  all  functional  requirements  and  the  initial  and  allocated 
baselines  for  development,  test,  and  deployment  have  been  established. 
Combining  these  events  allows  PMW  240  to  shorten  the  data  collection  and 
review  timeline  to  only  pertinent  information. 

•  Test  Readiness  Review  (TRR) — The  Mobility  APM-E  conducts  this  review 
once  the  application’s  initial  development  effort  has  been  completed.  This 
review  will  assess  the  application’s  readiness  to  begin  initial  formal  testing 
procedures.  These  procedures  include  testing  the  application  on  PMW  240- 
owned  mobile  devices  (including  both  smartphones  and  tablets),  as  well  as 
on  mobile  platform  simulator  software.  It  also  includes  conducting  needed 
security  scans. 

•  Production  Readiness  Review  (PRR) — The  Mobility  APM-E  and  Mobility 
PD  conduct  the  Production  Readiness  Review  (PRR)  following  the 
completion  of  initial  testing  on  PMW  240-owned  smartphones  and  tablets. 
This  review  will  analyze  the  application’s  readiness  to  begin  the  migration 
process  to  the  target  mobile  application  stores.  This  analysis  will  include 
further  testing  of  the  application  on  personally-owned  devices  to  ensure  the 
integrity  of  its  performance. 

The  PMW  240  Mobility  Project  Team  works  with  the  mobile  application  developer  to 
ensure  the  evaluation  criteria  for  entrance  and  exit  of  particular  technical  events  are 
appropriate  to  the  level  of  effort,  cost,  schedule,  and  complexity  of  the  mobile  application. 

The  240  mobility  project  team  reviews  developer-crafted  test  cases  to  ensure  they 
prove  completion  of  capabilities  they  are  written  to  test.  The  contractor  performs  a  final 
quality  assurance  testing  phase  after  the  final  development  iteration.  After  this  final  testing 
iteration,  the  government  enters  the  final  acceptance  testing  procedures.  The  PMW  240 
Mobility  Project  Team  also  integrates  security  and  usability  testing  and  evaluation  into  the 
development  process  to  streamline  testing  cycles  and  overall  impact  to  cost  and  schedule. 
The  developer  ensures  that  prior  to  each  iteration  release,  the  developed  application  has 
gone  through  a  cycle  of  developer  level  testing  to  ensure  functionality  intended  for 
demonstration  and  preview  is  functioning  as  expected. 
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During  the  government’s  acceptance  testing,  the  government  solicits  the  necessary 
testing  resources  that  align  with  the  application’s  target  audience,  and  the  Test  Team  lead 
verifies  all  application  functionality  works  as  designed. 

Final  application  approval  and  permission  to  release  the  application  is  at  the 
discretion  of  the  PMW  240  Mobility  Team  Decision  Authority  after  the  Development  Team 
adjudicates  and  addresses  submitted  defects  and  comments. 

Compared  to  normal  weapon  system  and  IT  SETR  events  and  timelines,  this 
process  tailored  for  PMW  240  mobile  applications  is  highly  innovative  and  extremely  fast  in 
delivering  capability  to  end-users  (PMW  240  Sea  Warrior  Program,  2015b). 

MPT&E  Mobile  Application  Toolbox 

In  an  effort  to  encourage  application  proliferation  for  the  Navy  MPT&E  community, 
PMW  240  developed  a  “toolbox”  that  is  accessible  to  civilian,  active  duty,  and  reserve 
members  of  the  Navy.  While  PMW  240  would  still  be  the  clearing  house  and  publishing 
authority,  the  toolkit  provides  tips,  style  guidance,  information  on  development 
environments,  and  tricks  for  building  specific  application  components  for  any  development 
audience  that  wishes  to  build  mobile  applications  that  look,  feel,  function,  and  are 
compatible  with  those  PMW  240  has  built  for  the  Navy  MPT&E  community.  The  toolbox  is 
designed  as  a  self-sustaining  wiki,  meaning  that  developers  can  use  the  site  to  post 
questions,  read  topic  forums,  and  even  contact  the  PMW  240  mobility  team  for  specific 
questions.  The  website  will  highlight  sample  projects  and  serve  as  an  additional  execution 
arm  to  the  work  being  completed  in  PMW  240.  The  use  of  a  toolkit,  shown  in  Figure  5, 
allows  developers  to  structure  their  application  to  appear  and  operate  as  an  official  U.S. 
Navy  application,  yet  encourages  development  by  third  parties.  It  is  an  extremely  innovative 
and  collaborative  approach  that  allows  any  capable  entity  to  develop  MPT&E  approved 
mobile  applications,  instead  of  restricting  that  ability  to  one  single  vendor  or  in-house 
Government  development  team  (PMW  240  Sea  Warrior  Program,  2016b). 


Figure  5.  MPT&E  Mobility  Toolkit 
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Sustainment  Phase  Innovations 

Although  the  Idea  phase  of  the  lifecycle  is  relatively  short  and  simple,  PMW  240  has 
applied  innovative  guidance  and  tools  to  both  accelerate  and  simplify  this  phase. 

Customized  Sustainment  Plans  and  Reviews 

Once  an  application  has  been  released  for  use,  PMW  240’s  primary  task  is  to  ensure 
the  content  and  technology  is  current,  functional,  and  accessible  to  the  user  base.  As 
discussed  previously,  each  Product  Owner  signs  a  POA  before  development  begins,  and 
the  POA  is  followed  throughout  sustainment  of  the  application.  On  a  quarterly  basis,  the 
application  is  reviewed  to  assess  the  value  of  the  investment.  Along  with  metrics  and 
feedback,  discussed  below,  the  internal  PMW  240  team  reviews  the  need  for  any  updates 
(either  content-related  or  technical)  to  determine  whether  an  application’s  state  is 
acceptable.  On  a  biannual  basis,  the  Product  Owners  are  invited  to  the  reviews  and  discuss 
the  feasibility  of  an  upgrade,  application  usage,  alignment  with  the  Product  Owner’s  team, 
and  to  decide  whether  the  application’s  status  warrants  continued  sustainment  funding. 
Monitoring  and  evaluating  the  applications  every  three  months  prevents  stagnant  content, 
unwarranted  investment,  and  insightful  trend  analysis,  and  fosters  the  relationship  with  the 
Product  Owners — all  on  an  innovatively  manageable  level  and  with  a  minimal  time 
investment. 

Metric  and  Feedback  Collection 

PMW  240  collects  metrics  and  feedback  from  each  of  the  applications  in  order  to 
better  assess  the  status  of  any  particular  application  and  use  the  results  to  determine 
continued  investment.  Metrics  are  aggregated  from  the  application  stores  and  a  built-in 
feedback  mechanism  within  the  application.  From  the  public  stores,  PMW  240  can  see  star 
ratings,  comments,  number  of  downloads  per  day,  and  devices  that  use  the  application. 

From  the  inherent  mobile  applications  feedback  mechanism,  PMW  240  can  view,  respond 
to,  and  route  comments  to  the  development  or  product  owner  team  for  consideration. 
Comments  are  often  in  the  form  of  suggestions  for  additional  content/functionality  or  reports 
of  bugs.  The  above-mentioned  feedback  is  collected  on  a  weekly  basis,  and  the  compiled 
version  is  distributed  on  a  monthly  basis  for  review.  The  PMW  240  team  can  actively  monitor 
an  application’s  usage,  end  user  reactions,  and  any  issues  and  incorporate  any  resulting 
changes  into  future  builds  of  the  application.  This  innovatively  thorough  yet  rapid  and  easy- 
to-decipher  data  collection  and  monitoring  allows  for  a  direct  feedback  loop  and  response 
adjudication  to  better  serve  the  end  user’s  needs. 

Challenges  and  Next  Steps 

Mobility  within  the  Navy  will  continue  to  grow  and  reach  a  broader  audience.  With 
this  growth,  the  demand  for  more  mobile  capabilities,  including  transactional  applications 
that  interact  with  current  DoD  systems,  will  increase.  There  are,  however,  a  number  of 
challenges  facing  the  Navy — particularly  in  the  use  case  of  BYOD  mobile  platforms — to 
ensure  its  workforce  can  fully  utilize  mobile  capabilities  for  all  their  mission  requirements. 
Some  of  these  challenges  include  using  Derived  Credentials,  opening  an  Official  Navy  App 
Store,  and  implementing  a  Mobile  Application  Management  (MAM)  framework. 

Derived  Credentials 

Supporting  secure  access  to  mobile  devices  through  ‘Derived  Credentials’  (a 
National  Institute  of  Science  and  Technology  coined  term  to  describe  cryptographic 
credentials  derived  from  Personal  Identity  Verification  [PIV]  and  Common  Access  Cards 
[CACs])  is  one  of  the  Navy’s  and  U.S.  Government’s  biggest  challenges  for  enabling  its 
mobile  workforce  to  securely  access  and  authenticate  mobile  devices  interacting  with 
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Government  data.  The  current  use  of  physical  CACs  and  card  readers  limits  the  use  cases 
of  usable  mobile  devices  and  is  un-scalable,  resulting  in  high  costs  to  implementation.  Using 
software,  micro-hardware,  or  other  cryptographic  methods  of  access  and  authentication  will 
have  to  be  developed,  tested,  and  put  into  production  before  the  Navy  fully  realizes  the  full 
suite  of  mobile  capabilities  currently  available  to  the  commercial  world. 

Navy  App  Store 

Providing  Sailors  and  civilians  access  to  a  full  suite  of  official  Navy  mobile 
applications,  designed  to  enable  their  day-to-day  work,  will  ensure  they  have  access  to 
officially  authorized  Navy  information  and  applications.  If  and  when  realized,  this  “Navy  App 
Store”  could  serve  both  GFE  and  BYOD  platforms/devices.  Establishing  this  app  store  will 
provide  a  single  location  for  Sailors  and  civilians  to  access  Navy  applications  and  content 
without  fear  of  downloading  a  fake  or  malicious  Navy  application  in  the  open  commercial 
app  stores. 

MAM  Framework 

Along  with  an  official  Navy  application  store,  utilizing  a  MAM  service  to  manage  the 
growing  number  of  Navy  applications  will  be  vital  to  sustainment.  Keeping  applications  up  to 
date  with  their  respective  operating  system  and  hardware  platforms,  as  well  as  content 
updates,  will  ensure  the  applications  end-users  will  have  fully  operational  apps  with  current 
information.  A  MAM  can  also  provide  robust  security  for  Navy  applications  when  loaded  to  a 
Sailor’s  personal  device  which  may  allow  for  transactional  applications  that  connect  securely 
with  DoD  systems  while  restricting  access  to  any  personal  data  on  the  device.  PMW  240  is 
currently  performing  a  Material  Solutions  Analysis  (MSA)  on  various  MAM  vendors  and  will 
assess  potential  application  use  cases  for  future  production. 

Conclusion 

PMW  240  has  developed  a  streamlined  and  agile  process  to  securely  acquire  and 
deliver  high  quality  mobile  MPT&E  applications  to  Sailors  and  civilians.  As  the  appetite  for 
mobile  applications  and  information  consumption  continues  to  grow,  PMW  240  will  continue 
to  be  flexible  and  scalable  with  its  acquisition  and  associated  mobile  application  fielding 
processes  to  meet  end  user  and  Department  of  the  Navy  future  needs  while  maintaining 
information  security  and  assurance  standards. 
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Chong  Wang,  Associate  Professor,  NPS 

Rene  Rendon,  Associate  Professor,  NPS 

1st  Lt  Crystal  Champion,  USAF 

Capt  Meredith  Ellen,  USAF 

Capt  Jenny  Walk,  USAF 

Freedom  of  Market  Navigation  Versus  Duty  of  Economic  Rescue:  The  U.S. 
Department  of  the  Navy’s  Use  of  Set-Asides  Parity,  Discretion,  and 

Simplified  Acquisitions  to  Contract  With  Hubzone  Small  Businesses 

Max  Kidalov,  Assistant  Professor,  NPS 

Elliott  Branch — is  the  Deputy  Assistant  Secretary  of  the  Navy  (Acquisition  and  Procurement)  in  the 
Office  of  the  Assistant  Secretary  of  the  Navy  (Research,  Development  &  Acquisition).  He  is  the  senior 
career  civilian  responsible  for  acquisition  and  contracting  policy  that  governs  the  operation  of  the 
Navy’s  worldwide,  multibillion-dollar  acquisition  system.  Branch  is  the  principal  civilian  advisor  to  the 
Navy  Acquisition  Executive,  serves  as  the  Department  of  the  Navy’s  Competition  Advocate  General 
for  procurement  matters,  and  is  the  community  leader  of  the  Navy’s  contracting  workforce.  Prior  to 
joining  the  Navy  Acquisition  Executive  staff,  Branch  was  the  first  civilian  director  of  contracts  at  the 
Naval  Sea  Systems  Command.  In  that  role,  he  led  one  of  the  largest  and  most  complex  procurement 
organizations  in  the  federal  government.  As  the  senior  civilian  for  contracting  at  NAVSEA,  Branch 
was  responsible  for  the  contractual  oversight  of  the  nation’s  most  complex  shipbuilding  and  weapons 
systems  procurement  programs.  His  duties  involved  the  obligation  and  expenditure  of  approximately 
$25  billion  annually.  Branch  is  a  member  of  the  Senior  Executive  Service  (SES).  Members  of  the  SES 
serve  in  the  key  positions  just  below  the  top  presidential  appointees.  They  are  the  major  link  between 
these  appointees  and  the  rest  of  the  federal  work  force.  SES  members  operate  and  oversee  nearly 
every  government  activity  in  approximately  75  agencies. 

Branch  spent  time  in  the  private  sector,  where  he  specialized  in  acquisition  and  project  management 
education,  training,  and  consulting  for  the  federal  workforce  and  its  associated  contractors.  In  this 
role,  Branch  was  responsible  for  the  design,  development,  delivery,  and  maintenance  for  a  wide 
variety  of  course  material  ranging  from  project  management  to  contract  law.  Branch’s  clients  included 
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the  Computer  Sciences  Corporation,  QSS  Group,  BAE  Systems,  the  Pension  Benefit  Guaranty 
Corporation,  and  the  Departments  of  Defense,  Energy,  Justice,  and  State. 

Prior  to  that,  he  served  as  the  chief  procurement  officer  for  the  government  of  the  District  of 
Columbia,  where  he  was  the  agency  head  responsible  for  procurement  operations,  for  policy,  and  for 
formulating  legislative  proposals  for  local  and  congressional  consideration.  Branch  led  a  staff  of  over 
200  employees  that  supported  over  40  city  agencies,  administered  a  $14  million  annual  operating 
budget,  and  oversaw  the  placement  of  $1 .5  billion,  annually,  in  city  contracts.  Before  joining  the 
District  government,  Branch  held  various  positions  in  the  SES  with  the  Department  of  the  Navy 
(DoN).  In  1993,  he  became  a  member  of  the  SES  as  the  director,  Shipbuilding  Contracts  Division,  at 
NAVSEA.  He  next  served  as  executive  director,  Acquisition  and  Business  Management  for  the  DoN, 
responsible  for  policy  and  oversight  of  contract  operations  throughout  the  entire  Navy.  While  in  this 
position,  he  also  served  as  project  executive  officer,  Acquisition  Related  Business  Systems.  In  this 
role,  he  was  responsible  for  the  formulation  and  execution  of  a  multi-year  effort  transforming  the 
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Abstract 

Strategic  sourcing  involves  aligning  the  processes  and  effects  of  the  purchasing  and  supply 
management  function  to  the  organization’s  overall  business  strategy.  Strategic  sourcing  aims 
to  add  value  to  the  organization  through  enhanced  supplier  relationships,  total  ownership  cost 
reduction,  and  demand  management.  In  the  Air  Force  (AF),  the  agency  charged  with 
implementing  strategic  sourcing  for  all  installation-level  spend  is  the  Air  Force  Installation 
Contracting  Agency  (AFICA).  AFICA  needed  a  way  to  determine  which  supplies  and  services 
represent  the  best  strategic  sourcing  opportunity — a  prioritization  model  that  “digs”  through 
the  mountain  of  spend  to  find  veins  of  “gold.” 

This  research  develops  a  spend  analysis  prioritization  model  that  mirrors  those  used  by  the 
commercial  sector.  It  marries  internal  AF  spend  data  to  external  market  data  to  gain  a 
comprehensive  view  of  each  supply  and  service,  and  its  potential  as  a  strategic  sourcing 
opportunity.  Ultimately,  1,706  supplies  and  services  are  ranked  based  on  their  strategic 
sourcing  opportunity  score,  thus  providing  a  guidepost  for  AFICA  to  assign  resources  to 
opportunities  with  the  most  potential  value.  Using  this  new  approach,  AFICA  can  combine 
supplies  and  services  into  related  categories  to  more  strategically  manage  related  spend, 
allowing  Category  Management  teams  to  thoroughly  understand  demand,  underlying  costs, 
and  the  market. 

Introduction 

The  Air  Force  Installation  Contracting  Agency  (AFICA)  is  tasked  with  “managing  and 
executing  above-Wing-level  operational  acquisition  solutions,  across  the  Air  Force 
enterprise”  (AFICA,  2016).  In  years  past,  strategic  sourcing  projects  were  selected  using 
pivot  tables  that  examined  the  attributes  of  each  federal  supply  code  (FSC)  or  product 
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service  code  (PSC).1  The  process  involved  examining  dollars  obligated,  number  of  contracts 
written,  number  of  suppliers,  and  other  basic  attributes  that  were  readily  available  in  the 
data.  Projects  were  also  selected  based  on  customer  demand,  meaning  that  if  a  customer 
felt  their  project  was  worthy  of  being  strategically  sourced,  AFICA  (and  its  predecessor 
organizations)  would  dedicate  a  team  to  investigate  the  potential  cost  and  process  savings 
associated  with  the  project  and  make  a  decision  to  proceed  or  not  based  on  those  potential 
savings.  This  process  is  labor-intensive,  and  AFICA  soon  realized  it  must  take  a  more 
proactive  approach  to  finding  new  strategic  sourcing  opportunities  in  order  to  more  easily 
find  the  veins  of  “gold”  hidden  in  their  “mountain”  of  spend. 

The  purpose  of  this  research  is  to  discuss  the  new  proactive  approach  that  was 
designed  as  a  collaborative  effort  between  AFICA/KA  (Strategic  Plans  and  Communication 
Directorate)  and  the  Naval  Postgraduate  School  (NPS).  This  new  approach  mirrors  the 
spend  analyses  that  have  been  performed  in  industry  for  decades.  It  marries  internal  Air 
Force  (AF)  spend  data  to  external  market  data  to  gain  a  comprehensive  view  of  each 
FSC/PSC  and  its  potential  for  strategic  sourcing.  Ultimately,  1 ,706  FSCs/PSCs  are  ranked 
based  on  their  opportunity  score,  thus  providing  a  guidepost  for  AFICA  to  assign  resources 
to  FSCs/PSCs  with  the  most  potential  value. 

Using  this  new  approach,  AFICA  can  combine  FSCs/PSCs  into  related  categories  in 
order  to  more  strategically  manage  AF  installation  spend.2  Those  categories  are  managed 
by  Category  Management  teams,  whose  primary  goal  is  to  thoroughly  understand  the 
demand,  the  underlying  costs,  and  the  market  related  to  the  category  in  order  to  properly 
manage  the  category’s  spend.  The  value  of  our  research  lies  in  understanding  which 
FSCs/PSCs  represent  the  best  strategic  sourcing  opportunities  for  the  AF  in  order  to 
properly  assign  limited  resources  to  exploit  potential  category  savings.  We  want  AF  “miners” 
to  “dig”  in  locations  with  the  highest  likelihood  of  “gold.” 

The  remainder  of  this  paper  proceeds  as  follows:  The  next  section  discusses  the 
growing  literature  related  to  strategic  sourcing  and  spend  analysis,  to  include  a  discussion  of 
the  government’s  strategic  sourcing  goals.  The  Methodology  section  details  the  methods  we 
used  to  create  and  implement  the  algorithm  that  prioritizes  the  PSCs.  The  Results  section 
provides  results  of  the  algorithm,  and  the  final  section  concludes  the  research  and  discusses 
next  steps. 

Literature  Review 

Purchasing  Transformation — Strategic  Sourcing  in  the  Commercial  Sector 

Transformation  in  the  purchasing  function  began  in  the  commercial  sector  in  the 
1990s.  As  the  business  world  became  more  global,  organizations  began  looking  for  new 
ways  to  not  only  compete  on  a  global  scale,  but  also  to  gain  competitive  advantages.  They 
soon  discovered  that  a  more  strategic  approach  to  managing  their  costs  and  supply  base 


1  “Also  referred  to  as  federal  supply  codes,  product  service  codes  are  used  by  the  United  States 
government  to  describe  the  products,  services,  and  research  and  development  purchased  by  the 
government”  (Outreach  Systems,  2016).  FSCs  describe  products,  while  PSCs  describe  services.  We 
examine  both  in  our  analysis. 

2  While  this  research  examines  all  AF  spend  data — installation  and  weapon  system — we  are  only 
interested  in  finding  strategic  sourcing  opportunities  in  the  installation  portion  of  the  data. 
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could  reap  huge  savings,  allowing  them  to  produce  at  lower  cost — and  often  better  quality — 
than  their  competitors.  Purchasing  moved  from  a  relatively  ignored  administrative  function, 
to  a  more  holistic  supply  management  function  that  aimed  to  cross  organizational 
boundaries  in  order  to  better  predict  supply  needs  and  deliver  better  quality  goods  and 
services.  The  tactical  function  of  purchasing  was  no  longer  useful  in  the  global  marketplace. 
In  its  stead  came  a  “more  transformational  process,  performed  at  higher  organizational  level 
...  [that]  examine[s]  the  whole  supply  network,  its  linkages,  and  how  they  impact 
procurement  and  purchasing  decisions”  (Wallace  &  Xia,  2014).  This  process  is  known  as 
strategic  sourcing. 

Strategic  sourcing  was  developed  to  better  align  the  mission  of  the  supply 
management  function  to  the  organization’s  overall  business  strategy.  In  short,  strategic 
sourcing  aims  to  add  value  to  the  organization  through  enhanced  supplier  relationships, 
reduced  total  ownership  costs,  and  demand  management.  Dwyer  and  Limberakis  (2011) 
identify  organizations  that  are  Best-in-Class  strategic  sourcing  performers  using  the 
following  criteria:  (1 )  spend  under  the  management  of  the  procurement  group,  (2) 
procurement  contract  compliance,  and  (3)  realized/implemented  cost  savings  (p.  2).  In  their 
study  of  315  companies  across  the  globe,  they  found  that  Best-in-Class  performers 
achieved 

•  37%  higher  spend  under  management 

•  72%  higher  contract  compliance 

•  52%  higher  realized/implemented  cost  savings 

Clearly,  implementing  strategic  sourcing  can  vastly  improve  the  purchasing  and 
supply  management  function.  So  why  haven’t  all  procurement  organizations  implemented  a 
strategic  sourcing  process?  Most  find  it  difficult  to  get  past  the  very  first  step. 

Spend  Analysis 

Laseter’s  (1998)  Balanced  Sourcing  Model  involves  seven  steps:  (1)  spend  analysis, 
(2)  industry  analysis,  (3)  cost/performance  analysis,  (4)  supplier  role  analysis,  (5)  business 
process  reintegration,  (6)  savings  quantification,  and  (7)  implementation.  This  research 
focuses  mainly  on  Step  1,  the  purpose  of  which  is  to  understand  the  organization’s  historical 
spend  patterns  by  examining  them  from  many  different  angles.  We  also  touch  briefly  on 
Step  2,  as  external  market  data  is  critical  to  developing  a  sound  sourcing  strategy.  Many 
regard  spend  analysis  as  the  most  difficult  step  in  the  strategic  sourcing  process  (RAND, 
2004;  Handfield,  2006;  Pandit  &  Marmanis,  2008).  It  takes  the  longest  amount  of  time  to 
implement  and  it  requires  a  team  of  persistent  researchers  who  are  willing  to  diligently  track 
down  the  disparate  data  required  to  truly  understand  the  organization’s  history  of  spend  (or 
“profile”)  in  the  category.  Even  after  the  data  are  aligned,  they  are  often  not  readily 
analyzable,  that  is,  they  require  a  large  amount  of  cleaning  to  achieve  accurate  results.  Take 
Handfield’s  (2006)  bleak  assessment: 

Be  careful!  Doing  a  spend  analysis  can  in  some  cases  mean  diving  into  a 
black  hole.  In  about  80  percent  of  the  companies  we  interviewed,  an  initial 
venture  into  spending  analysis  proved  to  be  a  data  nightmare.  For  example, 
many  companies  found  that  their  spend  analyses  were  tracked  using  Excel 
spreadsheets,  (p.  110) 

Despite  these  difficulties,  all  agree  that  the  spend  analysis  is  the  most  critical  step, 
as  all  subsequent  steps  rely  on  the  information  gathered  therein. 

Although  such  an  analysis  can  be  time-consuming  and  labor-intensive, 
private  enterprises  have  found  that  without  a  spend  analysis  it  is  difficult  to 
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identify  prospective  targets  for  applying  better  [purchasing  and  supply 
management]  practices,  develop  supply  strategies  for  specific  commodities, 
select  the  best  suppliers,  manage  suppliers  in  a  way  to  maximize  rewards 
and  minimize  risks,  and  convince  all  senior  leadership  of  the  need  to  shift  to 
best  [purchasing  and  supply  management]  practices  and  of  the  need  for 
resources  for  the  shift.  (RAND,  2004,  p.  7) 

Naturally,  an  organization  must  first  understand  its  history  of  spend  in  a  given  supply 
or  service  in  order  to  make  decisions  to  improve  sourcing.  “Spend  analysis  is  the  starting 
point  of  strategic  sourcing  and  creates  the  foundation  for  spend  visibility,  compliance,  and 
control”  (Pandit  &  Marmanis,  2008,  p.  5).  A  spend  analysis  “can  help  enterprises  improve 
their  purchasing  practices  in  the  areas  where  they  are  likely  to  produce  the  greatest 
benefits”  (RAND,  2004,  p.  vii).  Once  an  organization  understands  their  spend  history,  they 
can  develop  ways  to  reduce  or  aggregate  demand,  rationalize  suppliers  to  the  optimal 
number,  achieve  volume  discounts  by  leveraging  spend,  develop  methods  to  improve 
supplier  performance,  and  minimize  transaction  costs.  In  short,  strategic  sourcing  cannot 
happen  without  first  conducting  a  spend  analysis. 

A  spend  analysis  begins  with  the  collection  of  data.  For  most  organizations  new  to 
spend  analysis,  this  often  involves  consolidating  data  across  several  different  databases,  as 
few  organizations  have  their  data  organized  at  the  corporate  level.  Data  consolidation  is 
often  a  cumbersome  process — data  fields  do  not  match  perfectly,  making  them  difficult  to 
combine  into  a  rich  set  of  data  that  contains  all  the  information  needed  for  a  spend 
analysis.3  Once  the  data  have  been  collected  and  consolidated,  the  next  step  in  the  spend 
analysis  is  to  identify  opportunities  for  strategic  sourcing. 

Opportunity  Assessment 

Handfield  (2006)  defines  an  opportunity  as  supplies  or  services  that  have  “a 
reasonable  possibility  of  adding  more  value,”  with  value  coming  in  the  form  of  time,  money, 
and/or  quality  (p.  54).  Therefore  strategic  sourcing  opportunities  are  those  that  can  save  the 
organization  time  and/or  money  while  maintaining  or  increasing  quality. 

Spend-level  opportunities  can  be  identified  by  examining  as  many  of  the  following 
variables  as  possible  (Pandit  &  Marmanis,  2008):4 

•  Number  of  vendors  per  supply  or  service  (known  as  vendor  fragmentation) 

•  Number  of  purchasing  offices  per  supplier 

•  Number  of  contracts  across  purchasing  offices 

•  Number  of  purchases  from  preferred/non-preferred  suppliers 


3  The  ideal  spend  analysis  application  contains  components  that  allow  for  data  definition  and  loading, 
data  enrichment,  spend  data  analytics,  and  knowledgebase  management  (Pandit  &  Marmanis,  2008). 

4  Notably,  not  all  of  these  variables  are  available  in  AF’s  spend  data.  However,  many  of  these 
variables  were  used  in  the  RAND  (2004)  report  that  uses  spend  analysis  to  identify  strategic  sourcing 
opportunities  for  the  AF.  Their  report  focuses  on  all  AF  spend,  and  the  results  point  to  achieving  more 
value  in  weapon  systems  spend.  Our  research  focuses  on  where  to  achieve  more  value  using  only 
installation  spend  data. 
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•  Diversity  spend  compliance  (known  as  socio-economic  factors  in  the 
government) 

•  Amount  of  spend  with  suppliers  with  good  performance/bad  performance 

Each  of  the  variables  should  be  examined  for  each  supply  or  service.  Then,  once  the 
information  has  been  unraveled  at  the  lowest  possible  level,  aggregation  into  appropriate 
categories  can  occur.  Once  aggregated,  categories  can  be  scored  to  show  which  present 
the  best  opportunities  for  strategic  sourcing.  Clearly,  those  with  the  largest  potential  for 
savings  with  the  easiest  implementation  should  be  the  top  priority.  Pandit  and  Marmanis 
(2008,  p.  81)  use  an  “implementation  wave”  analogy  to  determine  which  opportunities  to 
address  first,  shown  in  Figure  1 . 
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Figure  1.  Opportunity  Implementation  Wave 

(Pandit  &  Marmanis,  2008) 

Opportunity  assessment  does  not  stop  after  all  the  internal  spend  analysis  has  been 
completed.  Instead,  the  internal  data  are  married  to  external  data  (Step  2  in  Laseter’s 
model)  that  addresses  the  market  conditions  associated  with  the  supply  or  service.  “A  spend 
analysis  integrates  internal  spend  data  and  external  supplier  and  market  data  and  applies 
analytical  techniques  to  help  identify  risks  and  opportunities  for  performance  improvements 
and  savings  by  applying  best  practices  in  purchasing  and  supply  management”  (RAND, 
2004,  p.  8).  Using  internal  and  external  data,  the  most  viable  strategic  sourcing  opportunities 
are  identified,  cross-functional  teams  are  created  to  further  develop  the  profile  of  the 
category  (i.e.,  develop  cost/time  savings  and  demand  management  estimates),  and  the 
process  continues  through  the  remainder  of  Laseter’s  (1998)  model. 

Purchasing  Transformation — Strategic  Sourcing  in  the  Federal  Government 

The  Federal  government  began  the  purchasing  transformation  in  the  early  2000s.  In 
2003,  then-principal  deputy  under  secretary  for  acquisition,  technology,  and  logistics 
(USD/AT&L),  Michael  W.  Wynne,  challenged  the  DoD  to  make  improvements  to  the 
acquisition  process  by  generating  value-added  changes  (Rendon,  2005).  In  2004,  then- 
director  of  defense  procurement  and  acquisition  policy,  Deidre  Lee,  noted  that  “strategic 
sourcing  and  commodity  councils  [are]  procurement  processes  that  are  designed  so  more 
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could  be  done  with  less  by  migrating  large  contracts  to  regional  centers  and  consolidating 
like  services”  (Rendon,  2005,  p.  13).  The  Office  of  Management  and  Budget  (OMB)  issued  a 
memorandum  to  all  Chief  Acquisition  Officers,  Chief  Financial  Officers,  and  Chief 
Information  Officers  to  “leverage  spending  to  the  maximum  extent  possible  through  strategic 
sourcing”  (OMB,  2005,  p.  1).  Agencies  were  expected  to  develop  strategic  sourcing 
governance,  goals  and  objectives,  performance  measures,  and  communication  and  training 
strategies  to  begin  implementing  strategic  sourcing. 

In  response  to  these  calls  to  action,  the  AF  began  the  process  of  strategic  sourcing 
in  2003  with  the  advent  of  the  Information  Technology  Commodity  Council.  This  commodity 
council  was  charged  with  standardizing  the  computers  available  to  AF  units  while  reducing 
spend.  To  do  that,  they  developed  three  computer  configurations  that  were  available  for 
purchase  and  negotiated  a  deal  with  Dell  Computers  for  12,500  computers.  The  savings 
from  this  deal  allowed  the  AF  to  purchase  2,500  more  computers  than  planned  for  in  the 
initial  procurement  (Rendon,  2005). 

With  its  first  success  under  its  belt,  the  AF  created  the  Enterprise  Sourcing  Squadron 
(ESS)  in  2010.  Along  with  other  responsibilities,  the  squadron  was  tasked  with  finding  more 
opportunities  for  strategic  sourcing.  ESS  later  became  the  Enterprise  Sourcing  Group  (ESG) 
and  in  2013,  the  AFICA.  During  that  timeframe,  the  OMB  issued  another  memorandum  that 
provided  more  detailed  strategic  sourcing  guidance  to  the  agencies,  including  the 
designation  of  Strategic  Sourcing  Accountable  Officials,  an  Interagency  Strategic  Sourcing 
Leadership  Council,  and  identification  of  the  characteristics  that  all  government-wide 
strategic  sourcing  vehicles  should  have  (OMB,  2012).  Using  this  guidance,  agencies  have 
been  working  hard  to  establish  their  strategic  sourcing  programs. 

The  AFICA  currently  has  six  commodity  councils  under  its  purview,  including: 
Information  Technology,  Medical  Services,  Furnishings,  Force  Protection,  Civil  Engineering, 
and  Knowledge  Based  Services  (U.S.  Air  Force,  n.d.).  While  these  councils  have  been 
successful  at  managing  demand,  reducing  costs,  and  improving  quality,  the  organization 
must  constantly  search  for  the  next  supply  or  service  to  strategically  source.  The  AF  is  a 
very  large  buyer,  purchasing  an  average  of  $59.8  billion  in  supplies  or  services  annually. 

A  2012  GAO  report  found  that  as  of  fiscal  year  201 1 ,  the  DoD,  the  Department  of 
Homeland  Security,  the  Department  of  Energy,  and  Veterans  Affairs — which  collectively 
account  for  80%  of  federal  procurement  spending — spent  only  5%  of  their  funding  using 
strategic  sourcing  techniques  (p.  7). 5  In  defense  of  these  organizations,  spend  analysis  is  a 
difficult  and  time-consuming  process,  made  worse  by  the  fact  that  the  data  required  to 
conduct  spend  analyses  exist  in  many  different  systems  that  are  not  linked  for  easy 
consolidation.  Further,  most  federal  agencies  lack  the  required  employee  expertise  to  lead 
strategic  sourcing  efforts. 

Despite  these  limitations,  the  AF  is  the  leading  strategic  sourcing  organization  in  the 
DoD  (Defense  Procurement  &  Acquisition  Policy,  n.d.).  Recognizing  the  need  to  identify 
installation-level  strategic  sourcing  opportunities  by  conducting  a  thorough  spend  analysis, 


5  The  DoD  was  just  slightly  better  than  the  average  among  the  four  departments,  with  5.8%  of  spend 
via  strategic  sourcing.  AF  efforts  in  FY1 1  account  for  3.7%  of  spend  via  strategic  sourcing,  which  is 
higher  than  any  other  military  service  component. 
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AFICA  partnered  with  NPS  to  develop  a  strategic  sourcing  prioritization  model.  We  detail  the 
methods  used  to  develop  and  implement  the  model  in  the  next  section. 

Methodology 

Data 

Our  analysis  uses  five  years  of  data  (FY2010-FY2014)  from  the  Federal 
Procurement  Data  System-Next  Generation  (FPDS-NG).  Data  held  in  this  system  are  input 
via  DD350s-lndividual  Contract  Action  Reports  (CARs),  and  consist  of  699,522  cases.6 

Although  the  data  were  readily  available  to  us,  they  did  not  come  without  limitations. 
AF  spend  data  is  not  particularly  “clean” — there  are  known  problems  related  to  user  input. 
The  system  relies  on  input  from  several  thousand  users — that  alone  increases  the  potential 
for  input  error.  Further,  many  of  those  doing  the  inputting  do  not  know  what  ultimately 
happens  with  the  data,  therefore  they  have  little  incentive  to  be  perfectly  correct  with  their 
input. 

Another  limitation  of  the  data  is  that  it  is  not  as  comprehensive  as  we  would  like  it  to 
be,  another  typical  problem  with  research-related  empirical  data.  FPDS-NG  does  not 
currently  capture  all  the  fields  needed  to  perform  the  most  rigorous  spend  analysis  possible. 
See  RAND  (2004)  for  a  detailed  assessment  of  the  issues  related  to  AF  spend  data. 

While  we  recognize  our  data  are  not  perfect,  using  it  is  far  better  than  continuing  to 
rely  on  a  reactive  approach  to  strategic  sourcing.  Thus  we  proceeded  with  the  research, 
which  involved  two  overarching  steps:  (1)  creating  the  prioritization  algorithm  that  uses 
internal  spend  data  to  determine  which  FSCs/PSCs  have  the  most  potential  for  strategic 
sourcing,  and  (2)  matching  the  related  external  market  data  to  those  FSCs/PSCs  to  further 
assess  strategic  sourcing  viability. 

Prioritization  Model 

The  prioritization  model  was  created  by  (1 )  culling  the  data  for  variables  most  useful 
for  conducting  a  spend  analysis,  and  (2)  assigning  weights  to  select  for  the  variables  we  feel 
are  most  important.  We  discuss  each  of  these  steps  in  detail. 

Selecting  Variables 

FPDS-NG  contains  more  than  250  variables.  To  be  parsimonious,  we  trimmed  the 
number  of  factors  to  the  seven  available  in  the  data  that  are  most  similar  to  those  used  by 
the  private  sectors  to  perform  their  spend  analyses,  and  those  we  believe  have  the  highest 
reliability.7  Those  seven  variables  are:  (1)  number  of  contracts,  (2)  number  of  suppliers,  (3) 
number  of  purchasing  offices,  (4)  number  of  offers  received,  (5)  total  obligated  dollar 
amount,  (6)  contracts  per  time  period,  and  (7)  number  of  AF  major  commands  (MAJCOMs) 
that  purchased  the  supply  or  service. 

The  first  variable,  number  of  contracts,  assesses  how  many  times  in  the  last  five 
years  a  contract  action  has  been  performed  to  purchase  the  supply  or  service.  The  larger 


6  In  this  case,  a  case  is  a  contract  action. 

7  In  this  case,  reliability  refers  to  the  likelihood  that  the  data  were  input  correctly — that  the  user  filled 
out  the  DD350  correctly  and/or  that  the  system  generates  the  input  automatically,  thus  reducing  input 
error. 
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the  number  of  contracts,  the  higher  potential  that  a  strategic  sourcing  opportunity  exists  to 
gain  volume  discounts  and  reduce  transaction  costs  by  consolidating  purchases. 

The  second  variable,  number  of  suppliers,  assesses  how  many  suppliers  the  AF 
uses  to  purchase  the  supply  or  service.  The  larger  the  number  of  suppliers,  the  higher 
potential  that  a  strategic  sourcing  opportunity  exists,  as  strategic  sourcing  involves 
rationalizing  the  supply  base  to  the  appropriate  number  of  suppliers  to  match  the  value  and 
risk  profile  for  the  supply  or  service.8 

Number  of  purchasing  offices,  the  third  variable,  assesses  how  many  different 
contracting  organizations  purchased  the  supply  or  service  over  the  last  five  years — it 
assesses  the  commonality  of  the  requirement.  The  larger  the  number  of  purchasing  offices, 
the  higher  potential  that  a  strategic  sourcing  opportunity  exists,  as  consolidating  purchases 
for  the  supply  or  service  allows  the  AF  to  leverage  their  strength  as  an  enterprise  (e.g., 
volume  discounts,  valuable  customer  benefits,  etc.),  rather  than  appearing  as  dozens  of 
smaller  customers  (i.e.,  individual  purchasing  offices)  to  the  suppliers. 

The  fourth  variable,  number  of  offers,  assesses  the  level  of  competition  received  in 
the  last  five  years.  The  larger  the  number  of  offers,  the  higher  competition  there  appears  to 
be  in  the  market.  Higher  competition  indicates  the  buyer  has  more  power  over  suppliers, 
which  equates  to  higher  potential  that  a  strategic  sourcing  opportunity  exists. 

The  fifth  variable,  total  obligated  dollar  amount,  is  a  simple  additive  total  of  the  spend 
for  each  FSC/PSC  over  the  last  five  years.  Naturally,  the  more  the  AF  spends  on  a  particular 
supply  or  service,  the  more  interested  the  organization  is  in  getting  that  spend  under 
management,  i.e.,  the  more  interested  they  are  in  strategically  sourcing  the  supply  or 
service  to  reap  cost  and  process  savings. 

The  sixth  variable,  contracts  per  time  period,  is  an  estimate  of  the  trend  in  purchases 
for  the  supply  or  service.  We  examine  whether  the  number  of  contracts  is  increasing, 
decreasing,  or  remaining  relatively  unchanged  over  the  five-year  period.  Clearly,  an 
increasing  trend  indicates  that  the  AF  should  consider  strategically  sourcing  the  supply  or 
service.  Unlike  the  other  variables,  this  variable  received  binary  coding,  where  an  increasing 
trend  received  a  score  of  1 ,  and  a  decreasing  or  unchanged  trend  received  a  score  of  0. 

Finally,  the  seventh  variable,  number  of  MAJCOMs  that  purchased  the  supply  or 
service,  assesses  the  universality  of  the  supply  or  service.  In  other  words,  are  all  MAJCOMs 
purchasing  the  supply  or  service,  or  is  it  only  being  purchased  by  a  certain  subset  of  the 
MAJCOMs?  For  instance,  consider  transient  alert  services.  This  service  is  only  used  for 
bases  with  flight  lines,  which  may  limit  the  MAJCOMs  who  purchase  the  service  to  Air 
Combat  Command,  Air  Mobility  Command,  and  Air  Force  Education  and  Training 
Command.  Using  this  variable,  we  are  attempting  to  assess  if  strategically  sourcing  the 


8  We  recognize  that  an  extremely  small  number  of  suppliers  is  also  cause  to  strategically  source,  as 
the  AF  would  benefit  from  developing  closer  relationships  with  critical  suppliers  who  have  little 
competition  in  their  markets.  In  our  data,  we  found  that  FSCs/PSCs  with  extremely  small  numbers  of 
suppliers  related  mostly  to  weapon  systems  spend.  Because  we  are  focusing  on  installation-level 
spend,  we  assume  that  the  larger  the  number  of  suppliers,  the  better  potential  strategic  sourcing 
opportunity. 
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supply  or  service  should  be  handled  at  the  enterprise-level  (AFICA)  or  at  the  MAJCOM- 
level. 


Before  weighting,  our  prioritization  algorithm  is  given  in  Equation  1. 

FSC /PSC  Score  =  ttContracts  +  # Suppliers  +  ttPurchOffices  +  ttOffers  + 
%Obligated  +  Trend  +  #MAJCOM  (1) 


Weighting  Variables 

We  recognize  that  each  variable  does  not  have  equal  influence  in  determining  the 
overall  prioritization  score  for  the  PSC.  Some  variables  matter  more  than  others.  We  used  a 
group  of  subject  matter  experts  to  discuss  and  assign  weights  to  each  variable.  After 
weighting,  our  prioritization  algorithm  is  given  in  Equation  2. 

FSC /PSC  Score  =  .20 (# Contracts)  +  .20 (# Suppliers')  +  20(#PurchOff  ices')  + 
.15  (#0ffers)  +  .Unobligated)  +  .08(Trend)  +  .05(#MAJCOM)  (2) 

When  summed,  the  weights  total  to  1.00,  or,  in  terms  of  percentages,  100%.  Number 
of  contracts,  number  of  suppliers,  and  number  of  purchasing  offices  are  the  highest 
weighted  variables,  each  receiving  a  weight  of  .20,  or  20%.  The  subject  matter  experts 
weighted  these  variables  heaviest  because  they  are  common  variables  used  by  industry 
experts  to  examine  commercial  spend  for  strategic  sourcing  opportunities.  Number  of  offers 
received  a  weight  of  .15,  or  15%.  Existence  of  competition  is  important  as  it  signals  the  AF’s 
ability  to  leverage  its  large  buying  power  to  get  a  better  deal. 

Total  dollars  obligated  received  a  weight  of  .12,  or  12%.  Some  readers  might  find  it 
odd  that  total  spend  for  the  supply  or  service  received  a  relatively  smaller  weight.  This 
decision  was  made  purposefully,  in  recognition  that  high  spend  does  not  necessarily  mean 
higher  potential  for  a  strategic  sourcing.  Some  high-spend  categories  may  already  be 
operating  on  thin  margins — the  savings  have  already  been  sifted  out.  It  is  important  to  note 
that  we  do  not  simply  ignore  high-spend  supplies  and  services.  We  take  special  measure  to 
include  those  FSCs/PSCs  in  the  opportunity  assessment,  which  is  discussed  later. 

The  lowest  weighted  categories  took  the  least  precedence  for  identifying  a  potential 
strategic  sourcing  opportunity  in  the  eyes  of  our  subject  matter  experts.  Trend  received  a 
weight  of  .08,  or  8%,  and  number  of  MAJCOMs  received  a  weight  of  .05,  or  5%. 

Applying  the  Weights 

Each  FSC/PSC  was  given  a  point  score  on  each  variable  that  could  not  exceed  the 
weight.  In  other  words,  the  FSC/PSC  with  the  largest  number  of  contracts  received  the  full 
weight:  .20.  The  FSC/PSC  with  the  next  larger  number  of  contracts  received  less  than  the 
full  weight,  a  decrease  equal  to  the  proportional  decrease  in  the  number  of  contracts.  For 
example,  FSC  7030,  ADP  Software,  had  the  highest  number  of  contracts.  It  received  a 
score  of  .20  points.  The  next  highest  number  of  contracts  belongs  to  FSC  7110,  Office 
Furniture.  It  received  a  score  of  .1209  points.  This  scoring  method  was  performed  for  each 
variable,  with  the  total  points  available  for  each  variable  equal  to  the  weight  for  the  variable. 

If  a  FSC/PSC  were  to  score  the  max  on  each  of  the  variables,  its  overall  score  would  be 
1 .00;  thus  the  closer  to  1 .00  in  overall  score,  the  higher  potential  that  a  strategic  sourcing 
opportunity  exists. 

Once  the  weights  were  applied  to  each  variable,  the  points  were  summed  for  each 
FSC/PSC,  creating  a  total  score  for  each  FSC/PSC.  Those  FSCs/PSCs  with  the  highest 
scores  are  considered  the  highest  potential  strategic  sourcing  opportunities.  Finally,  external 
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market  data  were  matched  with  the  highest  scoring  FSCs/PSCs  to  complete  the  spend 
analysis. 


Results 


We  examined  the  weighted  total  scores  from  two  different  angles:  total  score  and 
total  spend.  Using  this  method,  we  capture  FSCs/PSCs  that  scored  highly  using  the 
algorithm  as  well  as  FSCs/PSCs  that  may  not  have  scored  highly  in  the  algorithm,  but 
represent  a  substantial  amount  of  spend.  This  method  simultaneously  recognizes  that  spend 
may  not  necessarily  be  the  most  important  variable  (hence  the  lower  weighting  in  the 
algorithm),  but  it  is  still  an  important  factor  in  spend  analysis. 

First,  with  the  FSCs/PSCs  ranked  by  total  score,  we  asked  the  following  questions: 

•  Which  FSCs/PSCs  have  the  highest  overall  scores? 

•  How  much  spend  is  accounted  for  in  the  top  100  FSCs/PSCs? 

•  How  many  FSCs/PSCs  account  for  80%  of  the  total  spend? 

Table  1  shows  the  top  40  FSCs/PSCs  based  on  overall  score.  The  top  100 
FSCs/PSCs  account  for  64%  of  total  spend.  When  ranked  by  algorithm  score,  it  takes  281 
FSCs/PSCs  to  account  for  80%  of  the  total  spend.  Second,  with  the  FSCs/PSCs  ranked  by 
total  obligation,  we  asked  the  following  questions: 

•  Which  FSCs/PSCs  have  the  highest  total  obligation  (spend)? 

•  How  many  FSCs/PSCs  account  for  80%  of  the  total  spend? 

Table  2  shows  the  top  40  FSCs/PSCs  based  on  total  obligation.  The  top  67 
FSCs/PSCs  account  for  80%  of  the  total  spend. 

Because  we  wanted  to  focus  on  a  smaller  subset  of  FSCs/PSCs  (not  the  full  1 ,706 
FSCs/PSCs),  we  chose  to  use  the  67  FSCs/PSCs  that  account  for  80%  of  the  spend.  We 
selected  the  top  67  FSCs/PSCs  based  on  total  algorithm  score  and  the  top  67  FSCs/PSCs 
based  on  total  spend  score.  Thirty-two  of  the  FSCs/PSCs  were  duplicates — there  were  a 
total  of  102  unique  FSCs/PSCs  that  fell  into  the  top  67  in  each  category  (total  algorithm 
score  and  total  spend). 

Next,  we  performed  an  analysis  to  see  how  many  FSCs/PSCs  scored  in  the  top  67 
across  the  algorithm  variables.  We  sorted  the  data  by  each  algorithm  variable  and  selected 
the  top  67  FSCs/PSCs  for  each  variable.  Note  that  we  did  not  include  two  variables  in  this 
analysis:  Trend  and  Number  of  MAJCOMs.  These  variables  do  not  discriminate  well  across 
FSCs/PSCs,  so  they  were  not  useful  in  this  analysis.  When  lined  up  next  to  each  other,  we 
were  able  to  determine  how  many  times  a  specific  FSC/PSC  scored  in  the  top  67.  Naturally, 
the  more  times  an  FSC/PSC  scored  “high”  (i.e.,  in  the  top  67),  the  more  potential  it  has  as  a 
strategic  sourcing  opportunity.  See  Table  3  for  the  results.  Table  3  highlights  how  many 
times  an  FSC/PSC  scored  in  the  top  67.  The  table  shows  many  blue-  and  green-shaded 
FSCs/PSCs.  That  indicates  that  many  FSCs/PSCs  scored  in  the  top  67  across  four  or  more 
variables  in  the  algorithm.  That  is  a  positive  indication  that  the  model  is  identifying  the 
FSCs/PSCs  with  the  highest  potential  for  strategic  sourcing. 
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Table  3.  Top  67  FSCs/PSCs  for  Each  Variable 
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Within  Table  3,  there  are  145  individual  installation-level  FSCs/PSCs.9  We  separate 
those  145  FSCs/PSCs  into  two  categories:  Winners  and  Weirdos.  Winners  are  those 
FSCs/PSCs  that  had  a  high  total  score  from  the  algorithm  (top  67  on  algorithm)  and  scored 
in  the  top  67  across  three  or  more  different  algorithm  variables.  Clearly,  these  FSCs/PSCs 
present  a  high  likelihood  of  successful  strategic  sourcing,  thus  they  are  considered  Winners. 
There  are  45  installation-level  FSCs/PSCs  considered  Winners. 

The  FSCs/PSCs  in  the  second  category  are  considered  Weirdos — they  are  not  clear 
Winners,  but  they  are  also  not  clear  losers.  These  FSCs/PSCs  require  further  investigation 
to  determine  if  they  should  be  elevated  to  the  Winner  category,  or  if  they  do  not  have  the 
potential  for  successful  strategic  sourcing  and  should  be  dropped  from  analysis.  There  are 
two  ways  a  FSC/PSC  could  be  considered  a  Weirdo:  (1)  the  FSC/PSC  scored  in  the  top  67 
in  total  algorithm  score,  but  scored  in  the  top  67  in  just  two  (or  fewer)  algorithm  variables,  or 
(2)  the  FSC/PSC  was  in  the  top  67  of  overall  spend,  but  did  not  score  in  the  top  67  in  total 
algorithm  score.  There  are  71  installation-level  FSCs/PSCs  considered  Weirdos. 

After  separating  the  FSCs/PSCs  into  Winners  and  Weirdos,  we  added  an 
assessment  of  the  market  for  each  FSC/PSC  by  using  the  IBIS  Buyer  Power  score. 
IBISWorld  publishes  business  intelligence  reports,  including  detailed  reports  of  industries 
and  procurement  reports  that  provide  information  about  the  market,  average  purchase 
prices,  trends  in  the  market,  buyer  power  in  relation  to  the  market,  etc.  For  this  research,  we 
are  particularly  interested  in  their  procurement  reports,  specifically  the  buyer  power  score. 
IBIS  measures  buyer  power  based  on  a  weighted  average  of  Price  Trend,  Market  Structure, 
and  Market  Risk.  It  is  an  aggregated  measure  of  the  softness  of  the  market,  where  a  score 
of  1  means  the  supplier  has  more  power  and  a  score  of  5  means  the  buyer  has  more  power. 
The  average  score  for  our  FSCs/PSCs  was  3.48.  The  FSCs/PSCs  were  ranked  according  to 
buyer  power  score,  where  1  =  highest  buyer  power.10  Total  algorithm  score  ranks  were  then 
added  to  buyer  power  score  ranks  to  compute  a  Total  Rank  Score  for  each  FSC/PSC.  Thus, 
each  FSC’s/PSC’s  Total  Rank  Score  is  equal  to  their  internal  AF  rank  (using  the  total 
algorithm  score)  plus  their  external  market  rank  (using  the  buyer  power  score).  Naturally,  the 
lower  the  Total  Rank  Score,  the  more  potential  opportunity  exists  to  strategically  source  the 
FSC/PSC.  See  Table  4  for  a  list  of  FSCs/PSCs  ordered  by  Total  Rank  Score. 


9  These  results  show  29  FSCs/PSCs  that  belong  to  the  Air  Force  Sustainment  Center  (AFSC).  While 
not  carried  forward  in  this  analysis,  they  represent  potential  strategic  sourcing  opportunities  for  the 
AFSC. 

10  The  median  rank  was  53,  thus  a  rank  of  53  was  assigned  to  all  FSCs/PSCs  that  did  not  have  a 
corresponding  IBIS  Report. 
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Table  4.  Installation-Level  Winners  and  Weirdos — Total  Rank  Score 
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Finally,  using  the  OMB  taxonomy  of  categories  (Rung  &  Sharpe,  2015),  the 
FSCs/PSCs  were  summed  into  their  respective  categories.  Categories  then  received  an 
average  rank  score  (an  average  rank  score  of  all  FSCs/PSCs  included  in  the  category),  and 
the  categories  were  ranked  according  to  total  average  rank  score  and  total  spend.  Naturally, 
those  categories  with  the  lowest  rank  score  and  highest  amount  of  spend  represent  the  best 
potential  category  strategic  sourcing  opportunities.  See  Table  5  for  the  results  by  category. 


Table  5.  Installation-Level  Winners  and  Weirdos — Total  Rank  Score  by  Category 


Conclusion 

Our  results  suggest  that  Logistics  Support  Services,  IT  Hardware,  and  Business 
Administration  Services  had  the  best  Total  Rank  Score  and  would  likely  make  good  strategic 
sourcing  candidates.  Further,  large  total  obligation  categories  like  Management  Advisory 
Services,  Technical  and  Engineering  Services  (non-IT),  and  Facility  Related  Services 
account  for  nearly  one-third  of  all  the  spend — these  categories  would  also  make  good 
strategic  sourcing  candidates.  More  research  into  each  category  is  needed  to  assess  the 
true  viability  of  the  category  (i.e.,  estimated  cost  and  process  savings  and  how  demand 
management  might  affect  the  category). 

Spend  analysis  is  just  the  first  step  in  strategic  sourcing.  While  we  identify  categories 
that  represent  a  higher  likelihood  of  strategic  sourcing  success  in  this  research,  our  results 
are  based  solely  on  the  available  data — they  do  not  take  into  account  the  often-richer  data 
found  via  qualitative  analysis.  RAND  (2004)  warns  that  the  data  collected  via  the  DD350 
(CAR)  can  help  identify  potential  strategic  sourcing  opportunities,  “but  they  should  not  be 
used  to  make  final  decisions  to  develop  specific  supply  strategies  without  additional  data 
validation,  cleaning,  enhancement,  and  analyses  by  substantive  experts  and  manual 
resolution  of  anomalies”  (p.  15).  We  agree  with  this  assessment,  and,  to  that  end,  AFICA 
has  a  process  in  place  to  assign  Category  Management  teams  the  task  of  digging  deeper 
into  the  details  of  each  category  and  sub-category  of  spend  to  verify  if  savings  exist,  where 
specifically  those  savings  can  be  garnered,  and  how  to  adjust  policies  and  practices  to 
realize  those  savings  and  better  manage  consumption  (demand). 

AFICA  plans  to  profile  each  category  using  Category  Intelligence  Reports  (CIRs). 
Category  teams  are  tasked  with  completing  four  steps  to  confirm  and  estimate  potential 
savings.  After  the  spend  analysis  is  complete,  they  (1)  work  with  the  customer  to  identify  and 
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leverage  any  existing  customer  data  in  order  to  better  understand  the  demand  patterns  and 
potential  of  the  category  for  strategic  sourcing,  (2)  perform  a  more  in-depth  market  analysis 
to  understand  the  processes  of  the  commercial  sector  and  how  they  might  apply  to  the  AF, 
(3)  perform  a  gap  analysis  that  estimates  where  AF  processes  are  different  from  commercial 
processes,  and  how  to  minimize  the  gap  to  better  align  the  AF’s  practices  to  those  of  the 
commercial  sector  (when  beneficial),  and  (4)  develop  courses  of  action  to  present  to 
leadership  (i.e.,  AFICA  leadership  and  customer  leadership),  who  then  decide  whether  to 
proceed  with  strategic  sourcing,  and,  if  so,  which  course  of  action  to  use. 

In  summary,  the  goal  of  this  research  was  to  develop  a  prioritization  list  of  AF 
strategic  sourcing  opportunities  using  available  internal  spend  and  external  market  data.  We 
aimed  to  develop  an  easily  repeatable  process  that  quickly  enables  AFICA  “miners”  to  find 
the  “gold”  in  their  mountain  of  spend.  The  algorithm  we  developed  mirrors  those  used  in  the 
commercial  sector  and  can  be  used  by  other  service  components  to  quickly  identify  and 
prioritize  their  strategic  sourcing  opportunities. 
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Abstract 

The  Truth  in  Negotiations  Act  (TINA)  requires  contractors  (often  sole-source)  to  submit  “cost 
or  pricing  data”  that  is  “current,  complete,  and  accurate.”  The  intention  of  TINA  is  to  protect 
the  government  and  taxpayers  from  being  ripped  off  by  better  informed  contractors.  We 
argue  that  the  current  TINA  practice,  despite  its  good  intention,  is  subject  to  many  unintended 
negative  consequences  that  arise  from  contractors’  bad  incentives.  We  employ  an  incentive¬ 
centric  approach  to  perform  an  economic  analysis  of  TINA.  Our  analysis  indicates  that  the 
main  flaw  of  TINA  is  its  failure  to  address  the  moral  hazard  problem,  that  is,  contractors  lack 
proper  incentives  to  exert  their  best  efforts  to  achieve  cost  efficiency.  For  example,  in  fixed- 
price  contracts,  where  moral  hazard  is  otherwise  appropriately  addressed,  the  use  of  TINA 
undesirably  removes  contractors’  incentives  to  exert  effort.  The  policy  implication  of  this 
report  is  that  a  lax  use  of  TINA  in  the  context  of  firm-fixed-price  contracts  should  be  preferred 
to  a  strict  use.  Moreover,  in  a  repeated  game  situation  where  a  continuous  long-term  demand 
for  the  product  from  the  DoD  is  expected,  a  TINA  waiver  should  be  considered  for  the  early 
period  contracts  so  contractors  can  truthfully  reveal  their  best-effort  cost  information. 

Introduction 

The  Federal  Government  obligates  approximately  $500  billion  in  contracts  every  year 
for  supplies  and  services  needed  for  executing  its  mission  (Federal  Procurement  Data 
System-Next  Generation,  2015).  The  majority  of  procured  supplies  and  services  are  of  a 
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commercial  nature,  although  some  are  defense-unique  projects  for  research  and 
development  as  well  as  major  weapon  systems  acquisition.  Regardless  of  whether  the 
government  is  procuring  commercial-type  supplies  and  services  or  defense-unique  systems, 
the  government  aims  to  negotiate  a  fair  and  reasonable  price — fair  to  both  parties  and 
reasonable  considering  the  quality  and  timeliness  of  contract  performance  (FAR,  2015). 
When  procuring  supplies  and  services  readily  available  in  the  commercial  marketplace,  the 
government  relies  on  the  forces  of  market  competition  to  obtain  fair  and  reasonable  prices. 
However,  when  the  government  procures  defense-unique  supplies  and  services  in  markets 
where  there  may  be  limited  competition  or  only  one  seller,  the  government  relies  on 
statutory  requirements  to  ensure  a  level  playing  field  in  negotiating  fair  and  reasonable 
prices  with  contractors.  One  such  statute  is  the  Truth  in  Negotiations  Act  (TINA) 
promulgated  in  Public  Law  87-653.  TINA  was  enacted  to  enhance  the  government’s  ability 
to  negotiate  fair  and  reasonable  prices  by  ensuring  that  the  government  contracting  officer 
has  the  same  factual  information  that  is  available  to  the  contractor  at  the  time  of  price 
negotiations  (Nash  et  al.,  2007).  Advocates  of  TINA  argue  that  the  statute  effectively  levels 
the  playing  field  between  the  government  and  contractor  in  non-competitive  procurements, 
but  opponents  argue  that  TINA  is  not  only  administratively  burdensome,  but  also  results  in 
negative  unintended  consequences. 

The  purpose  of  this  research  is  to  analyze  the  Truth  in  Negotiations  Act  from  an 
economic  theory  perspective  focusing  on  contractor  incentives  under  different  contract 
types.  Our  research  question  asks  whether  TINA  provides  the  right  economic  incentive  to 
contractors  to  induce  their  best  effort  under  different  contract  types. 

The  remainder  of  the  paper  is  organized  as  follows:  The  following  section  provides  a 
detailed  description  of  the  Truth  in  Negotiations  Act.  After  that  is  a  review  of  economic 
literature  that  is  relevant  to  our  research  question.  Next  is  a  section  that  performs  analysis 
and  makes  policy  recommendations.  In  the  final  section,  we  conclude. 

Truth  in  Negotiations  Act 

Federal  acquisition  policy  requires  that  contracting  officers  procure  supplies  and 
services  from  responsible  sources  at  fair  and  reasonable  prices  (FAR,  2015).  Fair  and 
reasonable  prices  can  be  assured  through  the  use  of  competitive  proposals  providing  price 
competition,  commercial  or  catalog  prices,  or  prices  set  by  law  or  regulation  (FAR,  2015).  If 
these  approaches  are  not  available  in  a  procurement,  then  the  government  may  request  the 
offeror  to  provide  cost  or  pricing  data  to  be  used  in  negotiating  fair  and  reasonable  prices. 
Additionally,  the  offeror  may  be  required  to  certify  that  the  cost  or  pricing  data  provided  to 
the  government  are  current,  accurate,  and  complete  as  of  the  date  of  negotiations. 

During  the  procurement  planning  process,  the  government  will  conduct  requirements 
analysis  and  market  research  to  determine  the  availability  of  supplies  and  services  that  exist 
to  meet  the  government’s  requirements,  as  well  as  the  capability  of  the  market  to  provide 
those  supplies  and  services.  The  results  of  procurement  planning  will  determine  if  there  is  a 
competitive  market  for  the  required  supply  or  service.  Based  on  the  results  of  the 
procurement  planning  process,  the  government  will  conduct  solicitation  planning  and 
develop  the  solicitation  (e.g.,  a  Request  for  Proposal)  and  advertise  the  procurement 
opportunity  by  posting  the  solicitation  on  the  government-wide  electronic  portal. 

During  the  source  selection  process,  the  government  will  conduct  a  review  of  the 
proposals  and  determine  the  existence  of  adequate  price  competition,  commercial  or 
catalog  prices,  or  prices  set  by  law  or  regulation.  If  these  are  in  existence,  then  the 
government  will  be  able  to  conduct  a  price  analysis  on  the  proposals  and  there  will  be  no 
need  for  requiring  cost  or  pricing  data.  In  this  case,  the  TINA  requirements  will  not  apply. 
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If  adequate  price  competition,  commercial  or  catalog  prices,  or  prices  set  by  law  or 
regulation  are  not  in  existence — for  example,  if  only  one  proposal  is  received — then  the 
government  may  need  to  conduct  cost  analysis  as  part  of  the  evaluation  of  the  proposals. 
This  cost  analysis  may  require  the  offeror  to  provide  cost  and  pricing  data  to  the 
government.  The  FAR  (2015)  defines  cost  and  pricing  data  as  follows: 


Cost  or  pricing  data  (10  U.S.C.  2306a(h)(1)  and  41  U.S.C.  chapter  35)  means 
all  facts  that,  as  of  the  date  of  price  agreement,  or,  if  applicable,  an  earlier 
date  agreed  upon  between  the  parties  that  is  as  close  as  practicable  to  the 
date  of  agreement  on  price,  prudent  buyers  and  sellers  would  reasonably 
expect  to  affect  price  negotiations  significantly.  Cost  or  pricing  data  are 
factual,  not  judgmental;  and  are  verifiable.  While  they  do  not  indicate  the 
accuracy  of  the  prospective  contractor’s  judgment  about  estimated  future 
costs  or  projections,  they  do  include  the  data  forming  the  basis  for  that 
judgment.  Cost  or  pricing  data  are  more  than  historical  accounting  data;  they 
are  all  the  facts  that  can  be  reasonably  expected  to  contribute  to  the 
soundness  of  estimates  of  future  costs  and  to  the  validity  of  determinations  of 
costs  already  incurred.  They  also  include,  but  are  not  limited  to,  such  factors 
as — 

(1 )  Vendor  quotations; 

(2)  Nonrecurring  costs; 

(3)  Information  on  changes  in  production  methods  and  in  production  or 

purchasing  volume; 

(4)  Data  supporting  projections  of  business  prospects  and  objectives 

and  related  operations  costs; 

(5)  Unit-cost  trends  such  as  those  associated  with  labor  efficiency; 

(6)  Make-or-buy  decisions; 

(7)  Estimated  resources  to  attain  business  goals;  and 

(8)  Information  on  management  decisions  that  could  have  a  significant 

bearing  on  costs. 


Additionally,  if  the  value  of  the  procurement  exceeds  the  TINA  threshold  (currently 
established  at  $700,000),  the  offerors  will  be  required  to  certify  that  the  cost  or  pricing  data 
are  current,  accurate,  and  complete  at  the  time  of  negotiations.  This  is  the  essence  of  the 
TINA  requirement.  TINA  (10  U.S.C.  2306a  and  41  U.S.C.  ch.  35)  requires  offerors  to  submit 
certified  cost  or  pricing  data  if  a  procurement  exceeds  the  TINA  threshold  and  none  of  the 
exceptions  to  certified  cost  or  pricing  data  requirements  applies  (see  FAR  15.403).  Under 
TINA,  the  contracting  officer  obtains  accurate,  complete,  and  current  data  from  offerors  to 
establish  a  fair  and  reasonable  price  (see  FAR  15.403).  TINA  also  allows  for  a  price 
adjustment  remedy  if  it  is  later  found  that  a  contractor  did  not  provide  accurate,  complete, 
and  current  data. 

The  FAR  (2015)  defines  certified  cost  or  pricing  data  as  follows: 


Certified  cost  or  pricing  data  means  “cost  or  pricing  data”  that  were  required 
to  be  submitted  in  accordance  with  FAR  15.403-4  and  15.403-5  and  have 
been  certified,  or  are  required  to  be  certified,  in  accordance  with  15.406-2. 
This  certification  states  that,  to  the  best  of  the  person’s  knowledge  and  belief, 
the  cost  or  pricing  data  are  accurate,  complete,  and  current  as  of  a  date 
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certain  before  contract  award.  Cost  or  pricing  data  are  required  to  be  certified 
in  certain  procurements  (10  U.S.C.  2306a  and  41  U.S.C.  chapter  35). 

Thus,  during  the  source  selection  phase  of  contract  management,  in  situations  where 
the  government  does  not  have  adequate  price  competition,  commercial  or  catalog  prices,  or 
prices  set  by  law  or  regulation,  the  government  relies  on  the  contractor’s  certified  cost  or 
pricing  data  to  negotiate  a  fair  and  reasonable  price.  Once  negotiations  are  complete,  the 
contract  is  awarded.  The  contract  may  be  awarded  using  a  fixed-price  contract  or  a  cost- 
reimbursement  contract.  Fixed-price  types  of  contracts  provide  for  a  firm  price  or,  in 
appropriate  cases,  an  adjustable  price.  Fixed-price  contracts  providing  for  an  adjustable 
price  may  include  a  ceiling  price,  a  target  price  (including  target  cost),  or  both.  Unless 
otherwise  specified  in  the  contract,  the  ceiling  price  or  target  price  is  subject  to  adjustment 
only  by  operation  of  contract  clauses  providing  for  equitable  adjustment  or  other  revision  of 
the  contract  price  under  stated  circumstances.  Cost-reimbursement  types  of  contracts 
provide  for  payment  of  allowable  incurred  costs,  to  the  extent  prescribed  in  the  contract. 
These  contracts  establish  an  estimate  of  total  cost  for  the  purpose  of  obligating  funds  and 
establishing  a  ceiling  that  the  contractor  may  not  exceed  (except  at  its  own  risk)  without  the 
approval  of  the  contracting  officer  (FAR,  2015).  If,  during  contract  performance  or  even  after 
the  contract  is  complete,  the  government  determines  that  the  contractor’s  cost  or  pricing 
data  was  not  current,  accurate,  or  complete,  TINA  allows  for  a  price  adjustment  remedy  and 
can  recoup  any  excess  costs. 

During  the  contract  administration  phase  of  the  contract,  there  may  be  instances 
when  the  government  must  modify  the  requirements  of  the  contract.  Through  the  contract 
changes  process,  the  government  may  make  changes  within  the  general  scope  of  the 
contract  to  drawings,  designs,  or  specifications,  method  of  shipment  or  packing,  or  place  of 
delivery  (FAR,  2015).  Additionally,  if  any  such  change  causes  an  increase  or  decrease  in 
the  cost  of  any  part  of  the  work  under  the  contract,  the  government  will  negotiate  an 
equitable  adjustment  in  the  contract  price  and  modify  the  contract.  Since  this  contract 
change  will  occur  after  the  award  of  the  basic  contract,  the  government  will  not  have  the 
benefits  of  adequate  price  competition  in  determining  a  fair  and  reasonable  price.  Thus  the 
government  will  need  to  rely  on  the  contractor  to  submit  cost  and  pricing  data  to  the 
government,  and,  if  the  value  of  the  contract  change  exceeds  the  TINA  threshold  (currently 
established  at  $700,000),  the  contractor  will  be  required  to  certify  that  the  cost  or  pricing 
data  are  current,  accurate,  and  complete  at  the  time  of  contract  change  negotiation. 

When  the  contract  period  of  performance  is  over  and  the  completed  contract  is  being 
closed  out,  the  contractor’s  final  actual  costs  may  be  audited  by  the  government.  If  the 
government  has  reason  to  believe  that  the  contractor’s  certified  cost  or  pricing  data  was  not 
current,  accurate,  or  complete,  TINA  allows  for  a  price  adjustment  remedy  and  the 
government  can  recoup  any  excess  costs. 

As  can  be  seen  in  the  previous  discussion,  the  TINA  statute  is  integrated  throughout 
the  contract  management  process  and  provides  the  government  a  level  playing  field  with  the 
contractor  in  negotiating  a  fair  and  reasonable  price  without  the  benefit  of  price  of 
competition.  In  these  situations  the  government  and  contractor  may  be  negotiating  either  a 
fixed-price  contract  or  a  cost-reimbursement  contract.  The  next  section  will  discuss  the 
application  of  economic  theories  when  the  TINA  statute  is  used  in  each  of  these  contract 
type  categories. 

Economic  Literature  Review 

This  section  reviews  academic  literature  that  is  relevant  to  DoD  acquisition  and  sets 
a  foundation  for  the  subsequent  analyses.  We  first  start  with  a  general  description  of  the 
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unique  characteristics  that  underlie  DoD  major  weapon  system  acquisitions,  then  introduce 
adverse  selection  and  moral  hazard  concepts.  We  further  elaborate  on  why  DoD  contracting 
is  subject  to  both  adverse  selection  and  moral  hazard  problems,  and  consequently,  limiting 
information  rents  and  inducing  the  best  effort  naturally  become  the  two  objectives  for  policy 
makers.  We  also  introduce  the  concept  of  “power  of  incentive  schemes”  and  how  this 
concept  applies  to  various  contract  types.  Finally,  the  non-commitment  and  ratchet  effect  in 
DoD  contracting  is  discussed,  along  with  a  brief  introduction  to  the  cost  padding  behavior  of 
DoD  contractors. 

Unique  Characteristics  of  Major  Weapon  System  Acquisitions 

Major  weapon  system  purchases  are  very  unique  and  complicated.  Wang  and  San 
Miguel  (2013)  argue  that  “the  Major  Defense  Acquisition  Programs  (MDAP)  contracting 
environment  is  unique  in  the  sense  that  an  MDAP  contract  is  typically  a  sole-buyer-and- 
sole-seller  case,  in  which  market  competitive  forces  rarely  exist  and  significant  information 
asymmetry  and  potential  agency  problems  prevail”  (p.  6).  The  major  contributing  factor  to 
the  “sole  source”  or  “near  sole  source”  contracting  scenario  is  “the  complexity,  uncertainty, 
and  long-term  commitment  in  major  weapon  systems.”  Other  reasons  are  “the  DoD’s  need 
for  secrecy,  expediency,  and/or  safeguarding  human  resources”  (Wang  &  San  Miguel, 

2013). 

The  “sole-source”  scenario  puts  the  DoD  at  an  informational  disadvantageous 
position  relative  to  the  contractor  in  the  contracting  process.  Due  to  the  significant 
information  gap  between  the  contractor  and  the  government,  the  contractor  has  the  intent 
and  ability  to  extract  information  rents  from  the  government.  Moreover,  since  the  effort  level 
of  the  only  capable  contractor  is  not  observable,  contractors’  shirking  becomes  a  legitimate 
concern. 

Adverse  Selection  and  Moral  Hazard 

An  adverse  selection  (i.e.,  hidden  information)  problem  arises  when  contractors  have 
superior  information  relative  to  the  government.  Many  times,  the  government  is  at  a  loss 
when  it  comes  to  how  much  a  product  or  a  new  system  should  cost.  The  company  that 
provides  the  quote  is  at  a  high  advantage  when  it  negotiates  with  the  less  informed 
government.  The  government  usually  has  to  take  the  contractor’s  word  on  price  and  quality, 
especially  for  a  first-time-purchased  product  or  system. 

Laffont  and  Tirole  (1993)  provide  a  footnote  from  Robert  Keller,  who  was  the  former 
assistant  comptroller  general  of  the  United  States  in  regards  to  adverse  selection,  stating, 

The  government  negotiator  generally  is  at  a  disadvantage  in  trying  to 
negotiate,  since  the  contractor  knows  not  only  all  the  facts  and  the 
assumptions  underlying  his  estimates,  the  alternatives  available  to  him,  and 
the  contingent  areas,  but  he  also  knows  the  price  at  which  he  will  be  willing  to 
accept  the  contract,  (p.  2) 

Laffont  and  Tirole  (1993)  define  moral  hazard  (i.e.,  hidden  effort)  as  “endogenous 
variables  that  are  not  observed  by  the  regulator.  The  firm  takes  discretionary  actions  that 
affect  its  cost  or  the  quality  of  its  products.  The  generic  label  for  such  discretionary  actions  is 
effort”  (p.  1).  Effort  is  hard  to  observe  and  hence  cannot  be  contracted  upon.  As  a  whole, 
society  is  lazy  and  hence  contractors  tend  to  shirk  unless  incentives  are  provided  to  induce 
more  effort.  With  moral  hazard,  the  information  provided  by  the  contractors  on  their  past 
performance  and  quality  of  work  can  be  manipulated  to  make  it  seem  as  though  the 
company  is  making  their  best  effort,  and  some  very  well  might  be,  but  in  reality,  the 
contractors  are  shirking. 
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In  general,  DoD  contracts  are  subject  to  both  adverse  selection  and  moral  hazard 
problems,  given  that  significant  information  asymmetry  is  the  norm,  and  the  effort  levels  of 
contractors  are  generally  not  observable.  Hence,  a  benevolent  government  that  aims  to 
maximize  the  whole  society’s  welfare  has  two  policy  objectives  in  mind:  limiting  undue 
information  rents  and  inducing  cost-saving  effort. 

Various  Contract  Types  and  Power  Incentive  Schemes 

Fixed-Price,  Cost-Plus,  and  Incentive  Contracts 

There  are  two  major  types  of  contracts:  fixed-price  and  cost-plus  contracts.  The  two 
polar  cases  are  firm-fixed-price  (FFP)  and  cost-plus-fixed-fee  contracts  (CPFF). 

According  to  FAR  16.202-1, 

A  firm-fixed-price  contract  provides  for  a  price  that  is  not  subject  to  any 
adjustment  on  the  basis  of  the  contractor’s  cost  experience  in  performing  the 
contract.  This  contract  type  places  upon  the  contractor  maximum  risk  and  full 
responsibility  for  all  costs  and  resulting  profit  or  loss.  It  provides  maximum 
incentive  for  the  contractor  to  control  costs  and  perform  effectively  and 
imposes  a  minimum  administrative  burden  upon  the  contracting  parties. 

FAR  16.306  states, 

A  cost-plus-fixed-fee  contract  is  a  cost-reimbursement  contract  that  provides 
for  payment  to  the  contractor  of  a  negotiated  fee  that  is  fixed  at  the  inception 
of  the  contract.  The  fixed  fee  does  not  vary  with  actual  cost,  but  may  be 
adjusted  as  a  result  of  changes  in  the  work  to  be  performed  under  the 
contract.  This  contract  type  permits  contracting  for  efforts  that  might 
otherwise  present  too  great  a  risk  to  contractors,  but  it  provides  the  contractor 
only  a  minimum  incentive  to  control  costs. 

An  FFP  contract  without  TINA  addresses  the  moral  hazard  problem  but  still  suffers 
from  adverse  selection.  In  this  type  of  contract,  the  contractor  is  motivated  to  exert  the  best 
effort  to  save  on  cost  and  maximize  profit.  Adverse  selection,  on  the  other  hand,  is  still  a 
major  problem  due  to  contractors’  strong  incentive  to  withhold  their  proprietary  information 
as  well  as  extract  information  rents.  Even  with  market  research  completed  by  contracting 
officers,  the  adverse  selection  problem  remains  a  significant  issue. 

A  CPFF  contract,  in  contrast,  addresses  the  adverse  selection  problem  better 
because  the  reimbursement  is  based  on  incurred  rather  than  projected  cost.  However,  moral 
hazard  becomes  the  main  worry  since  contractors  have  no  incentive  to  curb  costs.  The  lack 
of  incentive  to  curb  costs  is  due  to  the  fact  that  the  contractor’s  profit  is  fixed  and  any  cost 
savings  will  be  passed  on  to  the  government  as  opposed  to  the  contractor. 

In  addition  to  the  FFP  and  CPFF,  there  are  various  incentive  contracts  that  lie 
between  the  two  extreme  cases.  They  are  fixed-price-incentive-fee  (FPIF)  contracts,  cost- 
plus-incentive-fee  (CPIF)  contracts,  and  cost-plus-award-fee  (CPAF)  contracts.  These 
incentive  contracts  are  intermediate  contracting  arrangements  between  the  two  polar  types, 
and  they  typically  address  both  adverse  selection  and  moral  hazard  to  some  degree,  yet 
neither  effectively  enough. 

Power  of  Incentive  Schemes 

Various  types  of  contracts  introduced  in  the  first  section  of  this  paper  possess 
different  power  of  incentive  schemes.  Power,  in  relation  to  incentive  schemes,  means  the 
extent  to  which  the  scheme  can  motivate  effort.  Table  1  is  reproduced  from  Laffont  and 
Tirole  (1993). 
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Table  1.  Power  of  Commonly  Used  Incentive  Schemes 

(Laffont  &  Tirole,  1993,  p.  11) 


Transfer  Allowed? 

Power 

Yes 

(procurement,  most  public 
enterprises) 

No 

(most  private  regulated 
firms) 

Very  High 

Fixed  price  contracts 

Price  caps 

(firm  residual  claimant) 

Intermediate 

Incentive  contracts 

Incentive  regulation 

(cost  or  profit  sharing) 

Very  Low 

Cost-plus  contracts 

Cost-of-service  regulation 

(government  or  consumers 
residual  claimants) 

Note.  Highlighted  emphasis,  added  by  authors. 


Laffont  and  Tirole  explain  that  a  cost-plus  contract  has  the  government  pay  the 
contractor  its  realized  price  while  the  fixed-price  contract  has  a  set  limit  that  the  government 
will  pay  no  matter  what  performance  or  effort  is  executed.  They  also  explain  that  the 
incentive  contracts  have  the  government  and  the  contractors  share  the  realized  costs. 

With  a  fixed-price  contract,  contractors  usually  put  forth  the  most  amount  of  effort. 
Although  the  contractor  knows  they  will  receive  a  fixed  fee  for  their  product,  the  more  they 
save  on  the  cost,  the  more  profit  they  will  receive.  Thus,  fixed-price  contracts  are  high  power 
incentive  schemes. 

A  cost-plus  contract  gives  few  incentives  to  the  contractor  to  exert  effort  and  hence  is 
labeled  as  a  low  power  incentive  scheme.  Incentive  contracts,  as  intermediate 
arrangements  between  fixed-price  and  cost-plus  contracts,  are  intermediate  power  incentive 
schemes. 

Table  2  in  Laffont  and  Tirole’s  (1993)  A  Theory  of  Incentives  in  Procurement  and 
Regulation  shows  that  if  a  contract  is  fixed-price,  the  effort  is  induced  100%  (p.  40).  If  the 
contract  is  cost-plus,  the  effort  is  induced  at  0%. 

Non-Commitment  and  the  Ratchet  Effect 

In  DoD  contracting,  contracts  are  awarded  for  one  basic  year  with  priced  options  for 
additional  years.  This  is  known  as  multiple-year  contracting.  Another  approach  is  multi-year 
contracting.  Multi-year  contracting  is  an  annual  contract  that  is  awarded  each  year 
consecutively.  In  cost-based  requirements,  multiple-year  contracts  may  be  used  to  provide 
long-term  incentives  to  contractors  while  providing  a  reliable  contract  vehicle  for  recurring 
needs.  Awarding  multiple-year  contracts  ensure  that  the  short-term  contract  is  guaranteed 
and  option  years  are  written  in  the  contract  for  long-term  commitment.  The  risk  of  exercising 
options  is  still  present,  but  at  a  lesser  extent  so  as  to  incentivize  the  contractor  to  perform 
well  in  order  to  guarantee  an  additional  year.  Multiple-year  contracts  do  not  require 
congressional  approval  or  guarantee  of  funds  stability,  and  can  be  used  for  cost- 
reimbursement  type  contracts  and  fixed-price  type  contracts.  To  gain  a  better 
understanding,  Table  2  shows  an  example  of  the  difference  between  multiple-year 
contracting  and  multi-year  contracting. 
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Table  2.  Multi-Year  vs.  Multiple-Year  Contracting 

(O’Rourke  and  Schwartz,  2014) 


Multi-year 


Multiple  year 


Issue  one  or  more  contracts  for  each 
year’s  procurement  of  four  aircraft  After 
Congress  funds  the  procurement  of  the 
first  four  aircraft  in  FY2015.  DoD  would 
issue  one  or  more  contracts  for  those  four 
aircraft.  The  next  year,  after  Congress 
funds  the  procurement  of  the  next  four 
aircraft  in  FY2015,  DoD  would  issue  one 
or  more  contracts  for  those  four  aircraft 


Issue  one  contract  covering  all  20  aircraft 
to  be  procured  during  the  five-year  period 
FY2015-FY2019.  Contract  award  in 
FY2015,  at  the  beginning  of  the  five-year 
period,  following  congressional  approval  to 
use  MYP  for  the  program,  and 
congressional  appropriation  of  the  FY2015 
funding  for  the  program.  Implementation  of 
the  contract  over  the  next  four  years  would 
be  completed  by  obtaining  funding  for 
each  additional  FY 


Laffont  and  Tirole  (1993)  state, 

If  the  firm  performs  well  (produces  at  a  low  cost)  early  in  the  relationship,  the 
regulator  infers  that  the  technological  parameter  is  favorable  and  tries  to 
extract  the  firm’s  rents  by  being  more  demanding  during  the  regulatory 
review.  The  firm  has  thus  an  incentive  to  keep  a  low  profile  by  not  engaging 
in  much  cost-reducing  activity.  To  induce  the  firm  to  produce  at  a  low  cost 
when  efficient,  the  regulator  must  offer  it  a  generous  reward  for  good 
performance,  (p.  45) 

Stated  equivalently,  the  lack  of  the  commitment  from  the  government  naturally  leads 
to  contractors’  fear  of  being  “ratcheted  up”  if  they  reveal  their  lowest  possible  cost.  Being 
efficient  one  time  would  eliminate  their  future  rents.  Therefore,  unless  the  profit  from  a  one- 
year  contract  is  sufficiently  sizable,  contractors  would  choose  not  to  engage  in  cost-saving 
activities  as  much  as  they  can. 

The  cure  to  the  problem  above  is  straightforward.  Laffont  and  Tirole  (1993)  state, 

To  put  the  ratchet  effect  in  perspective,  recall  that,  if  the  two  parties  can 
commit  to  a  long-term  contract  at  the  beginning  of  their  relationship,  the 
regulator  optimally  commits  to  use  each  period  the  optimal  static  contract. 
That  is,  it  is  optimal  for  the  regulator  to  commit  not  to  exploit  the  information 
acquired  from  observing  the  firm’s  performance.  Commitment  is  crucial  for 
this  outcome  because  the  regulator  would  want  to  fully  extract  the  firm’s  rents 
from  the  second  period  on  after  the  firm  reveals  its  efficiency  in  the  first 
period,  (p.  376) 

Cosf  Padding 

Cost  padding,  if  not  detected  and  controlled  by  the  government,  adds  unnecessary 
cost  to  the  government.  Examples  of  cost  padding  include  but  are  not  limited  to,  incurring 
excessive  costs  to  the  government  such  as  leisurely  meetings,  first  class  travel,  and 
business  lunches.  Other  examples  are  shifting  overhead  costs  from  commercial  business  to 
government  contracts  and  engaging  in  various  bookkeeping  tricks  to  manipulate  costs.  The 
government  counters  contractor  cost  padding  by  requiring  certain  contractors  to  be  audited. 
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The  DCMA  has  a  systemic  operational  cycle  that  allows  monitoring  contractor  cost 
driving  contractor  performance.  In  the  Defense  Contract  Audit  Agency  (DCAA;  2014a) 
Contract  Audit  Manual  (CAM),  Chapter  9  discusses  Audit  of  Cost  Estimates  and  Cost 
Proposals.  Cost  padding  is  a  factor  in  labor  cost  data.  It  states, 

The  auditor  should  examine,  on  a  selective  basis  and  in  cooperation  with 
Government  technicians  ...  for  the  new  product.  When  appropriate, 
contractor  personnel  should  be  interviewed  to  ascertain  probable  significant 
changes  in  engineering  production  methods  and  the  effect  those  changes 
might  have  on  current  cost  data.  When  an  evaluation  indicates  that  significant 
technological  changes  have  occurred  since  the  cost  data  was  accumulated, 
adjustment  of  experienced  costs  is  necessary  before  projecting  the 
experience  cost  pattern.  (DCAA,  2014a) 

The  manual  further  explains  the  contractors  variances  of  direct  labor  cost  and 
illustrates  a  “guesstimate”  is  made  and  then  a  “padding”  is  added  to  protect  from  any 
unexplained  cost.  Through  the  bookkeeping  manipulations,  resulting  “guesstimates”  and 
subsequent  “padding,”  the  contractor  audit  becomes  a  significant  challenge  to  accurately 
appraise  the  extraneous  cost.  Cost  padding  is  viewed  as  being  more  prevalent  in  cost-plus 
contracts,  though  as  will  be  elaborated  later,  the  incentives  for  cost  padding  still  exist  under 
a  fixed-price  contract. 

Analysis  and  Policy  Implications 

As  pointed  out  in  the  literature  review  section,  defense  procurement  is  subject  to 
both  adverse  selection  and  moral  hazard  problems;  consequently,  limiting  information  rents 
and  promoting  contractors’  cost-saving  efforts  become  the  two  main  policy  objectives  for  the 
government. 

This  section  argues  that  TINA,  to  some  extent,  mitigates  the  adverse  selection 
problem  by  mandating  that  contractors  provide  certified  cost  and  pricing  data  that  are 
“current,  complete,  and  accurate”  and  legally  holding  them  accountable.  Hence  it  is  fair  to 
say  that  TINA  helps  policy  makers  achieve  one  of  their  two  policy  goals:  limiting  information 
rents. 


This  section,  however,  emphasizes  the  ineffectiveness  of  TINA.  In  particular,  building 
on  an  economic-based,  incentive-centric  approach  that  investigates  contractors’  incentives, 
we  argue  that  the  main  flaw  of  TINA  is  its  failure  to  address  the  moral  hazard  problem.  In 
some  cases,  such  as  cost-plus  contracts,  where  moral  hazard  is  an  inherent  concern  to 
begin  with,  TINA  fails  to  provide  remedies.  More  detrimentally,  in  other  cases  such  as  fixed- 
price  contracts,  where  moral  hazard  is  otherwise  appropriately  addressed,  the  use  of  TINA 
undesirably  removes  contractors’  incentives  to  exert  effort.  Therefore,  TINA,  in  the  context  of 
fixed-price  contracts,  is  the  problem  rather  than  the  solution. 

Based  on  our  arguments,  we  accordingly  make  policy  recommendations  at  the  end 
of  this  section. 

Distorted  Incentives:  Use  of  Tina  With  Firm-Fixed-Price  (FFP)  Contracts 

In  this  subsection,  we  express  our  greatest  concern  over  TINA.  That  is,  ill-fated 
incentives  are  created  if  TINA  is  used  with  an  FFP  contract.  In  the  following,  we  use  a  step- 
by-step  approach  to  illustrate  the  problem. 

Background 

There  is  a  current  policy  push  toward  more  use  of  FFP  contracts.  Since  2009, 
support  for  firm-fixed-price  contracts  has  been  steadily  increasing  in  order  to  limit 
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government  risk,  reduce  cost  overruns,  and  improve  contract  effectiveness  (Wang  &  Miguel, 
2013).  As  such,  there  has  also  been  a  strong  policy  push  towards  regulation  in  support  of 
fixed-price  contracts  to  be  a  fix-all  to  the  cost  overruns  that  the  DoD  faced  in  prior  years.  Top 
leaders,  including  President  Obama;  Robert  Gates,  former  Secretary  of  the  DoD;  and 
Ashton  Carter,  former  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics,  have  all  expressed  that  they  favored  the  use  of  more  FFP  contracts  in  DoD 
acquisition.  The  presidential  memorandum  issued  in  April  2009  explicitly  stated  that  “there 
shall  be  a  preference  for  fixed-price  type  contracts”  (Obama,  2009).  Consequently,  more 
and  more  DoD  contracts  prescribe  FFP. 

Given  the  more  frequent  use  of  FFP  in  the  DoD  procurement,  it  has  become 
increasingly  more  important  to  understand  how  contractors’  incentives  change  with  respect 
to  the  enforcement  of  TINA  within  FFP  contracts.  In  particular,  we  will  use  a  “without  and 
with”  approach  to  demonstrate  the  unintended  negative  consequences  of  bundling  TINA 
with  FFP  contracts. 

FFP  Contracts  Without  TINA,  Despite  Many  Weaknesses,  Are  Free  of  the  Moral 

Hazard  Problem 

Wang  and  San  Miguel  (2013)  challenge  the  wisdom  behind  policy-makers’  favor 
toward  FFP  contracts.  In  particular,  they  state,  “The  notion  that  fixed  price  contracts  are 
better  than  cost-plus  contracts  for  limiting  cost  overruns  is  misleading.”  The  article  further 
explains  that  FFP  contracts  may  in  fact  have  three  negative  consequences:  (1)  fixed-price 
contracts  provide  few  risk-sharing  benefits;  (2)  fixed-price  contracts  lead  to  higher 
government  payments;  and  (3)  unjustified  favor  toward  fixed-price  contracts  promotes 
inefficient  industry  structure. 

Nevertheless,  despite  the  problems  pointed  out  by  Wang  and  San  Miguel  (2013), 

FFP  contracts  do  have  one  appeal:  That  is,  an  FFP  contract  is  a  high  power  incentive 
scheme  that  effectively  motivates  contractors’  maximum  efforts.  Once  an  FFP  contract  is 
awarded,  the  contractor  relentlessly  seeks  to  reduce  cost  because  every  dollar  saved  on 
cost  will  directly  translate  into  profit.  Stated  equivalently,  contractors  under  FFP  contracts 
without  TINA  voluntarily  abstain  themselves  from  shirking,  that  is,  the  moral  hazard  is  not  a 
problem  at  all. 

FFP  Contracts,  With  TINA,  Lose  the  Last  Benefit  of  Being  a  High  Power 

Incentive  Scheme 

Since  most  DoD  weapon  procurement  FFP  contracts  exceed  the  TINA  threshold 
value,  unless  the  TINA  waiver  is  widely  applied,  FFP  contracts  without  TINA  are  exceptions 
rather  than  norms.  Hence,  it  is  important  to  understand  what  incentives  or  disincentives  are 
created  or  removed  if  TINA  is  bundled  with  an  FFP  contract. 

One  astute  observation  by  Rogerson  (1994)  is  that  “TINA  cannot  force  defense 
contractors  to  reveal  the  lowest  possible  cost  that  they  could  produce  at  if  they  exerted  an 
optimal  effort.  Rather,  it  essentially  tells  them  that  the  price  they  negotiate  must  be  close  to 
the  cost  they  actually  incur.” 

Therefore,  a  contractor  under  an  FFP  contract  that  is  subject  to  TINA  has  the 
following  ill  incentive:  The  fear  of  being  held  accountable  for  any  significant  unfavorable  cost 
discrepancy  (i.e.,  the  actual  incurred  cost  is  significantly  below  the  ex-ante  cost  estimate 
submitted  to  the  DoD  as  the  basis  for  contract  fixed-price)  would  strongly  motivate  the 
contractor  to  shirk  (i.e.,  reduce  cost-saving  effort)  or  even  engage  in  cost  padding  (e.g.,  by 
opportunistically  incurring  or  allocating  more  costs  to  the  government  contracts),  especially 
when  the  natural  state  turns  out  to  be  favorable. 
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In  the  situation  above,  shirking  becomes  a  dominant  strategy  because  working  hard 
introduces  disutility  to  the  contractor  with  the  additional  risk  of  being  penalized  by  TINA.  In 
the  case  of  a  very  favorable  natural  state  (i.e.,  if  every  exogenous  factor  turns  out  to  be 
good),  if  shirking  is  not  sufficient  to  bring  the  cost  close  enough  to  the  ex-ante  cost  estimate, 
the  contractor  will  engage  in  opportunistic  and  hard-to-detect  cost  padding  to  ensure  the 
reported  cost  is  trouble-free. 

To  recap,  TINA,  in  the  context  of  FFP  contracts,  removes  the  last  benefit  of  FFP 
contracts  and  literally  turned  a  high  power  incentive  scheme  to  a  low  power  one.  Here,  the 
moral  hazard  problem  is  reintroduced  by  the  misuse  of  TINA. 

A  Numerical  Example 

We  use  the  theoretical  framework  in  Laffont  and  Tirole  (1993)  to  set  up  a  numerical 
example  to  illustrate  the  point  made  in  prior  sections.  A  contractor’s  cost  function  is  specified 
as  follows: 


c  =  c(p,e)  (1) 

where  p  is  a  state  parameter  (e.g.,  technology)  and  e  is  the  effort.  One  can  interpret 
that  p  is  the  adverse  selection  parameter  and  represents  contractor’s  private  information, 
and  e  is  the  moral  hazard  parameter. 

Without  losing  generality,  assume  the  state  parameter  p  has  three  possible 
outcomes:  good,  neutral,  or  bad,  with  equally  likely  probability.  Moreover,  the  contractor  can 
choose  either  work  hard  (e  =  10)  or  shirk  (e  =  1 ). 

Imagine  the  cost  function  takes  the  following  form: 


Note  that  the  cost  increases  with  p  (so  p  is  an  inverse  indicator  of  state  parameter) 
and  decreases  with  e  (effort  reduces  cost). 


Case  (1)  Good  situation:  (P  =  10),  with  probability  1/3. 

10 

c  =  10  +  —  (2) 

e 

Case  (2)  Neutral  situation:  (p  =  20),  with  probability  1/3. 

10 

c  =  20  +  —  (3) 

e 

Case  (3)  Bad  situation:  (p  =  30),  with  probability  1/3. 

10 

c  =  30  +  —  (4) 

e 

It  is  reasonable  to  assume  that  the  contractor  knows  the  probability  distribution  of  the 
natural  state,  whereas  the  government  does  not  know.  We  also  assume  that  the  contractor’s 
negotiation  strategy  is  to  ensure  breakeven  even  in  the  bad  situation  and  he  or  she  can  still 
shirk.  So  the  contractor  will  submit  $40  as  the  cost  estimate  by  Equation  4,  and  the  less 
informed  government  would  most  likely  accept,  with  TINA  strings  attached,  stating  that  if  the 
incurred  cost  is  more  than  25%  lower  than  $40  (i.e.,  below  $30),  then  the  contractor  is 
subject  to  a  TINA  audit. 

Let’s  also  assume  that  this  is  a  one-time  static  game  in  which  no  further  contract  is 
possible.  The  contractor  tries  to  maximize  its  profit. 
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The  sequence  of  the  actions  is  as  follows:  The  contractor  submits  the  bidding  price, 
accepted  by  the  government,  which  attaches  TINA  to  the  FFP  contract.  Then  the  natural 
state  reveals,  the  contractor  chooses  effort,  and  finally  the  cost  is  incurred. 

If  a  bad  situation  happens,  the  contractor  will  choose  to  work  hard  (e  =  10),  so  the 
cost  is  $31  by  Equation  4,  a  TINA  audit  is  not  triggered,  and  the  contractor  earned  a  profit  of 
$9.  There  is  no  moral  hazard  problem. 

In  the  case  of  a  neutral  situation,  if  the  contractor  works  hard  (e  =  10),  his  or  her  cost 
would  be  $21  by  Equation  3,  which  is  good  in  the  absence  of  TINA,  yet  not  so  when  TINA  is 
in  place.  Because  any  cost  below  $30  would  trigger  a  TINA  audit.  The  contractor,  knowing 
this  risk,  would  choose  to  shirk  (e  =  1)  so  the  cost  would  be  $30  by  Equation  3,  which 
successfully  hides  the  contractor  under  the  radar  of  TINA.  Now  a  moral  hazard  problem  is 
created  by  TINA. 

What  if  the  most  favorable  natural  state  emerges?  In  that  case,  if  the  contractor 
works  hard,  he  or  she  will  incur  a  cost  of  $1 1  by  Equation  2,  which  is  going  to  raise  a  big  red 
flag  to  the  government.  Therefore,  the  contractor  is  going  to  shirk;  however,  because  the 
natural  state  turns  out  to  be  so  favorable,  even  shirking  is  not  enough  to  mute  the  alarm  of 
TINA.  (Note  that  shirking  in  case  1  would  yield  a  cost  of  $20,  which  is  below  the  audit 
threshold  value  of  $30,  and  hence  will  trigger  the  TINA  audit.)  So  what  would  the  contractor 
do  to  evade  the  TINA  investigation?  The  contractor  will  engage  in  “cost  padding”  and 
artificially  increase  the  reported  cost  to  at  least  $30,  so  he  or  she  will  not  get  into  trouble. 
Now,  TINA  not  only  created  the  moral  hazard  problem,  it  also  generated  bad  incentives  for 
defense  contractors  to  engage  in  unethical  and  opportunistic  “cost  padding.” 

Fixing  Incentives:  From  Static  to  Dynamic  Perspective 

One-Shot  Static  Game 

A  good  starting  point  is  a  static  situation  where  no  further  contract  is  possible.  Using 
the  numerical  example,  the  government  already  paid  $40,  because  the  contractor  can  avoid 
a  TINA  audit  in  all  three  possible  scenarios,  by  either  “shirking”  or  “cost  padding”  or  both, 
government  payment  becomes  fixed.  Therefore,  any  higher  profit  of  the  contractor  will  lead 
to  a  higher  social  welfare.  The  implication  is  straightforward:  In  order  to  correct  the  ill 
incentives  created  by  TINA  in  the  context  of  FFP,  policy  makers  need  to  undo  the  bundling, 
that  is,  remove  TINA  from  FFP,  so  the  FFP  is  back  to  a  high  power  incentive  scheme. 

Repeated  Game  With  Non-Commitment 

In  the  one-shot  static  game,  when  TINA  is  removed  from  an  FFP  contract,  the 
contractor  is  fully  motivated  to  exert  the  best  effort  to  maximize  profit.  Since  no  future 
contract  is  possible,  the  contractor  is  not  afraid  to  reveal  private  information  (i.e.,  the 
minimum  cost  that  can  be  achieved  through  the  best  effort)  because  there  is  no  possibility 
for  the  government  to  exploit  the  private  information  revealed  against  the  contractor  in  the 
future. 

However,  in  reality,  the  relationship  between  a  typical  contractor  and  the  government 
is  rarely  a  one-shot  game.  Rather,  it  is  better  characterized  as  a  repeated  game  with  non¬ 
commitment  from  the  government.  Typically  when  multiple-year  contracts  are  awarded,  the 
government  is  agreeing  to  a  single-year  term  contract  with  the  option  of  additional  years. 
Nearing  the  end  of  the  current  fiscal  year,  the  government  will  begin  the  process  of 
exercising  the  next  option  year.  This  decision  is  a  unilateral  process  that  a  contractor  may 
consider  as  non-commitment  and  in  return  may  be  apprehensive  to  share  true  cost  or 
pricing  data  for  fear  of  being  “ratcheted  up”  in  future  years. 
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Stated  equivalently,  in  a  repeated  game  where  contracts  have  one  base  year  and 
option  years  which  can  be  exercised  by  the  government,  a  simple  removal  of  TINA  from  a 
one-year  FFP  contract  may  not  be  sufficient  to  induce  the  contractor’s  best  effort.  The 
contractor  is  in  a  very  vulnerable  position  in  the  sense  that  if  he  or  she  chooses  to  reveal 
private  information  at  the  early  stage  of  the  game,  that  information  may  be  used  against  him 
or  her  later  so  no  future  information  rents  are  possible.  As  discussed  in  the  literature  review 
section,  contractors’  fears  of  being  “ratcheted  up”  by  the  government  motivates  them  to 
withhold  their  private  information  so  they  can  still  extract  information  rents  from  the 
government  in  later  periods.  To  recap,  a  simple  removal  of  TINA  from  a  one-year  FFP 
contract  tends  to  be  ineffective  in  addressing  the  moral  hazard  problem 

So  what  is  the  fix  of  the  lack  of  incentives?  If  a  one-year  FFP  contract  without  TINA 
is  not  enough  to  motivate,  the  government  should  consider  multiple-year  FFP  contracts 
without  TINA.  This  is  especially  useful  if  the  product  is  demanded  on  a  continuous  basis. 
The  idea  is:  make  the  reward  of  revealing  the  best-effort  cost  big  enough,  so  the  contractor 
voluntarily  tells  the  government  what  is  the  lowest  achievable  cost.  It  is  wise  to  let  the 
contractor  win  early,  win  big,  but  only  win  once.  The  government,  and  hence  the  taxpayers, 
win  in  the  long  run  and  win  even  bigger. 

Multiple-Years  Contracts:  Numerical  Example  Continued 

In  this  subsection,  we  extend  the  static,  one-shot  numerical  example  to  a  repeated 
game  case.  Under  some  reasonable  assumptions,  we  show  that  government  savings  can 
be  achieved  by  fixing  contractors’  incentives. 

Without  losing  generality,  assume  the  government  needs  to  order  this  product  every 
year  for  15  years.  If  each  year  TINA  is  attached  for  15  annual  contracts,  the  contractor  will 
always  choose  to  shirk1  or  “shirk  and  cost  padding”  in  order  to  avoid  the  TINA  audit,  as  well 
as  to  keep  the  information  rents  for  the  future.  Hence,  the  government  will  end  up  paying 
$600.  Alternatively,  if  TINA  is  removed  for  every  annual  contract  on  a  yearly  basis,  the  TINA 
concern  is  removed  for  that  year;  however,  the  contractor  still  worries  about  the 
consequence  of  revealing  the  lowest  possible  cost  under  the  maximum  effort  due  to  the 
non-commitment  nature  of  government  contracts.  One-year  increased  profit  due  to  effort  is 
meagerly  too  small  to  entice  the  contractor  to  give  up  their  future  information  rents.  Thus  the 
contractor  will  still  withhold  effort  and  choose  to  shirk. 

Without  losing  generality,  assume  that  a  five-year  FFP  contract  is  sufficient  to  induce 
the  contractor  to  exert  his  or  her  best  effort.  Therefore,  the  government  commits  to  pay  $40 
each  year  for  five  years  with  no  TINA  strings  attached.  With  this  commitment,  the  contractor 
is  fully  motivated  to  work  as  hard  as  possible  and  the  lowest  possible  cost  is  revealed  to  the 
government.  The  government,  which  observes  that  the  true  expected  lowest  possible  cost  is 

$21  (i.e.,  -*11+-*21+-*  31),  will  use  that  information  to  price  future  10-year  contracts. 

Under  the  assumption  that  a  10%  profit  is  allowable,  the  government  will  offer  $23.1 
($21*1.1)  annual  FFP  contract  for  the  remaining  10  years.  So  the  total  government  payment 
now  becomes  $40*5+$23.1*10  =  $431,  a  savings  of  $169  relative  to  the  original  situation. 


1  Note  that  in  contrast  to  the  one-shot  game,  the  contractor  chooses  to  shirk  even  in  the  bad  situation, 
due  to  the  concern  of  being  “ratcheted  up”  if  the  lowest  possible  cost  is  revealed. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-247- 


Note  that  if  the  time  span  is  longer,  say  25  years  as  opposed  to  15  years,  then  the 
government  savings  will  be  even  larger. 

TINA  Waivers:  A  Useful  Policy  Tool 

TINA  is  effective  in  deterring  outright  fraud  and  “defective  pricing,”  especially  on  the 
part  of  the  cost  that  is  verifiable.  Hence,  we  should  give  TINA  credit  for  doing  that  part  right. 
However,  TINA  is  much  less  effective  at  addressing  the  moral  hazard  problem,  where  one 
key  determinant  of  the  cost,  namely  effort,  is  unobservable,  unverifiable,  and  not 
contractible.  TINA  could  even  become  very  destructive  when  it  is  applied  to  an  FFP  contract 
setting,  as  shown  earlier. 

Fortunately,  lawmakers  do  allow  TINA  waivers  and  a  shrewd  utilization  of  that  tool  is 
essential  for  making  better  use  of  TINA.  One  of  the  justifications  for  a  TINA  waiver  is  that 
“there  are  demonstrated  benefits  to  granting  the  waiver.”  Our  analysis  in  this  section 
detailed  the  reasoning  for  the  use  of  TINA  waivers.  Based  on  our  analyses,  we  recommend 
the  following  policies  options: 

If  an  FFP  contract  is  negotiated  with  a  contractor  who  is  unlikely  to  have  a 
continuous  contracting  relationship  with  the  government  for  the  same  or  similar  products 
and  services,  then  a  waiver  of  TINA  should  be  applied.  However,  it  can  sometimes  be 
difficult  to  predict  the  future  of  non-continuous  relationships  until  after  the  first  year  of 
performance.  Additionally,  the  Federal  Acquisition  Regulation  allows  for  certain  TINA 
waivers  under  HCA  approval.2 

If  an  FFP  contract  is  negotiated  with  a  contractor  who  is  likely  to  continue  to  provide 
the  same  or  similar  product  to  the  government  for  years  to  come,  then  a  multiple-year  FFP 
contract,  without  TINA  provisions  on  defective  pricing  data,  should  be  offered  to  motivate  the 
contractor’s  best  effort.  Note  that  in  this  setting,  a  multiple-year  contract  is  needed. 

TINA  and  Cost-Plus  and  Incentive  Contracts 

TINA  is  less  damaging  when  it  is  bundled  with  cost-plus  contracts.  In  such  contracts, 
the  moral  hazard  is  an  inherent  concern  to  start  with;  TINA  does  not  introduce  the  problem, 
nor  does  it  solve  it.  Under  a  cost-plus  contract,  the  contractor  shirks  anyway,  regardless  of 
the  presence  of  TINA.  To  the  extent  that  the  total  realized  cost  is  auditable  while  the  various 
components  of  total  cost  are  not  (Lafond  &  Tirole,  1993),  “cost  padding”  would  still  be 
possible.  That  said,  TINA  does  make  the  verifiable  part  of  the  cost  more  credible,  and  also 
provides  disincentives  for  contractors  to  engage  in  outright  fraud  and  “defective  pricing” 
behavior. 

Incentive  contracts  are  basically  intermediate  arrangements  between  fixed-price  and 
cost-plus  contracts.  Hence,  similar  to  an  FFP  setting,  but  to  a  lesser  degree,  any  cost-saving 


2  Increasing  the  use  of  TINA  waivers  may  be  a  plausible  solution  if  reasonable  expectations  exist  that 
fair  and  reasonable  pricing  is  already  established.  For  example,  per  FAR  14.403-1  (c)(4),  the  HCA 
may  waive  the  requirement  for  contractors  (and  lower-tiered  subcontractors)  to  provide  certified  cost 
or  pricing  data  if  such  data  was  previously  submitted  and  is  updated.  Allowing  for  more  waivers  is  an 
“easy-fix”  to  lowering  defective  pricing  cases,  but  it  may  not  be  the  most  effective  in  reducing 
disincentives  attached  to  TINA.  Waiving  TINA  may  also  subject  the  government  to  information  rents 
that  were  previously  mitigated.  Simply  waiving  policy  when  a  need  for  it  still  exists  is,  in  and  of  itself, 
an  ineffective  policy  solution. 
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incentives  under  incentive  contracts  would  be  weakened  by  TINA.  The  government  may 
change  contract  vehicles  depending  on  the  lifecycle  of  the  acquisition  program,  and  it  is 
important  to  know  how  TINA  will  affect  contracts  within  each  milestone  of  a  program. 
Throughout  the  lifecycle  of  the  acquisition,  a  requirement  may  move  along  the  contract 
vehicle  spectrum  to  take  into  account  new  discoveries  and  established  requirements. 
Because  of  this,  TINA  should  also  be  a  living-breathing  provision  that  takes  into  account  the 
different  contract  vehicles  used  in  major  acquisition  rather  than  an  end-all  to  pricing 
uncertainty.  Because  there  are  certain  adverse  selection  issues  and  moral  hazards  that  are 
unique  to  differing  contract  types,  acquisition  personnel  will  need  to  be  aware  of  which 
disincentives  may  be  occurring  at  each  contracting  stage.  We  leave  this  to  our  future 
research. 

Conclusion 

It  has  been  more  than  50  years  since  TINA  was  first  enacted  in  1962.  In  a  nutshell, 
TINA  requires  contractors  (often  sole-source)  to  submit  “cost  or  pricing  data”  when  they 
negotiate  the  price  of  a  contract  with  the  federal  government.  The  contractors  must  certify 
that  the  information  they  provide  is  “current,  complete,  and  accurate.”  Failing  to  disclose 
truthful  information  could  lead  to  a  civil  or  criminal  investigation.  The  intention  of  TINA  is  to 
protect  the  government  and  taxpayers  from  being  ripped  off  by  better  informed  contractors. 

We  hopefully  have  convinced  our  readers  that  the  current  TINA  practice,  despite  its 
good  intention,  is  subject  to  unintended  negative  consequences  that  arise  from  contractors’ 
bad  incentives.  Such  bad  incentives  are  inherently  associated  with  the  current  TINA 
framework.  We  document  both  strengths  and  weaknesses  of  the  current  TINA  practice,  with 
an  emphasis  on  the  latter  and  in  turn  generate  corrective  policy  implications. 

One  major  contribution  of  our  study  is  to  introduce  an  economics-based,  incentive¬ 
centric  approach  that  focuses  on  the  investigation  of  agents’  (i.e.,  DoD  contractors’)  various 
incentives  that  are  generated  by  TINA.  This  approach,  in  our  opinion,  can  be  widely  applied 
to  many  issues  in  the  DoD  acquisition  environment.  The  importance  of  agents’  (in  our  case, 
DoD  contractors’)  incentive  issues  can  never  be  overstated  in  a  DoD  procurement  setting, 
as  testified  by  Rogerson  (1994): 

Defense  procurement  is  unique  among  regulated  industries  in  the  United 
States  in  that  economists  have  played  virtually  no  role  in  helping  shape  its 
regulatory  practices  and  institutions.  Perhaps  this  is  due  to  the  barrier  to  entry 
created  by  the  need  to  first  learn  about  procurement  practices  or  to  a 
lingering  distaste  for  military  matters  among  academics.  Whatever  the 
reason,  this  lack  of  economic  input  is  unfortunate,  because  many  of  the 
regulatory  and  policy  issues  in  defense  procurement  involve  the  types  of 
incentive  issues  [emphasis  added]  that  economists  are  very  good  at 
analyzing.  My  own  hope  is  that  economists  are  on  their  way  to  colonizing  this 
new  policy  frontier  and  that  some  of  the  ideas  discussed  in  this  article  will 
play  a  role  in  shaping  policy  debates  over  the  next  decade,  (p.  87) 
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Abstract 

Since  March  1999,  the  Department  of  the  Navy  (DoN)  has  had  an  important  shore  duty:  to 
exercise  its  purchasing  freedoms  and  powers  in  ways  that  will  channel  government  contracts 
to  small  business  concerns  (SBCs)  locating  and  hiring  in  Historically  Underutilized  Business 
Zones  (HUBZones).  Under  the  Small  Business  Act,  the  HUBZone  Program  provides  these 
firms  with  contracting  assistance,  notably,  competitive  and  sole  sources  set-asides.  In  2005, 
this  assistance  was  expanded  to  FAR  Part  13  Simplified  Acquisition  Procedures.  The  federal 
government  must  also  spend  at  least  3%  of  its  prime  contract  dollars  with  HUBZone  SBCs. 
During  fiscal  years  2006-2015,  DoN’s  HUBZone  goal  achievements  experienced  a  brief 
dramatic  growth  followed  by  a  full-circle  decline  to  the  decade-old  spending  and  percentile 
levels.  Academic  and  legal  policy  literature  offers  many  possible  reasons  for  this  unfortunate 
U-turn.  The  HUBZone  Program’s  design  was  often  criticized  on  the  grounds  that  it  impedes 
creating  and  finding  capable  and  eligible  firms;  that  it  may  reward  already-successful  or 
overly-costly  contractors;  that  it  hurts  the  government’s  ability  to  meet  its  duties  to  other 
socioeconomic  programs,  such  as  the  8(a)  Program;  that  it  is  not  strong  enough  to  match  the 
8(a)  Program;  and  that  it  introduces  undue  complexity  into  federal  procurement.  Beginning  in 
201 1-2012,  the  federal  HUBZone  Program  lost  about  30%  of  its  participants  to 
decertification.  Finally,  during  the  first  half  of  the  last  decade,  the  HUBZone  Program 
triggered  a  constitutional  stand-off  between  legislative,  executive,  and  judicial  branches  on 
the  Program’s  discretionary  parity  or  mandatory  precedence  over  the  8(a)  SDB,  WOSB,  and 
SDVOSB  programs.  The  DoN  initially  scored  a  judicial  victory  over  the  Small  Business 
Administration  (SBA)  in  favor  of  precedence,  but  changed  its  stance  three  years  later  with  the 
rest  of  the  executive  branch  in  favor  of  discretion  and  parity.  This  study  examines  possible 
root  causes  and  solutions  to  DoN’s  HUBZone  contracting  woes  through  the  use  of  the 
generally  accepted  Cohen-Eimicke  Contract  Management  Performance  Model. 
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Introduction 

Since  1 999, 1  the  Small  Business  Act  and  the  Small  Business  Administration  (SBA) 
imposed  on  the  U.S.  Department  of  the  Navy  (DoN;  the  parent  agency  of  the  U.S.  Navy  and 
the  U.S.  Marine  Corps)  a  shore  duty  to  render  economic  development  assistance  to  small 
business  concerns  located  in  so-called  Historically  Underutilized  Business  Zones 
(HUBZones)  by  means  of  government  contracts.  This  rescue  duty2  was  imposed  as  a 
condition  on  the  pre-existing  customary  legal  freedom  enjoyed  by  Navy  and  Marine  buyers 
to  navigate  the  market  and  the  industrial  base  looking  for  the  best  bargain.3  Similar  to  the 
seafarers’  rescue  duty,  this  new  economic  development  duty  is  based  on  the  rationale  that 
the  Navy  and  the  Marines  must  use  their  superior  power  (in  this  case,  buying  power)  to 
assist  struggling-area  firms  and  not  merely  wait  on  some  development  assistance  agencies 
and  experts  to  come  help  later  (Dilger,  2013).  This  rescue-on-shore  framework 
contemplates  that,  as  condition  of  assistance,  the  DoN  will  necessarily  receive  substantial 
benefits  from  the  HUBZone  firms  in  the  form  of  goods  and  services  to  meet  the  DoN’s 
requirements.  Indeed,  HUBZone  industrial  base  benefits  may  well  be  mission-critical  for  the 
DoN. 


The  Small  Business  Act  and  implementing  regulations  require  the  DoN  to  follow 
certain  designated  paths  to  support  the  federal  HUBZone  policy.  For  example,  the  DoN  is 
required  to  meet  its  HUBZone  contracting  goal,  negotiated  so  as  to  contribute  to  the  DoD’s 
and  government-wide  goal  of  spending  not  less  than  3%  of  total  procurement  spending  with 
HUBZone  small  business  concerns.4  Further,  the  HUBZone  statute  and  regulations  provide 
several  tools,  including  competitive  (currently  including  reserves)  set-asides,  sole  source 
set-asides  (up  to  certain  dollar  thresholds),  a  price  evaluation  preference  in  full  and  open 
competitions  applicable  against  other  than  small  business  concerns,  and  a  certified 
HUBZone  small  firms’  database.5  In  addition,  as  part  of  rulemaking  on  veterans  contracting, 
the  Federal  Acquisition  Regulation  (FAR)  Council  on  its  own  made  “improving  opportunities” 
for  HUBZone  firms  one  of  the  stated  purposes  of  FAR  Part  13,  Simplified  Acquisition 
Procedures  (Federal  Acquisition  Regulation  [FAR]  Council,  2005). 

This  paper  contains  the  preliminary  results  of  a  study  conducted  at  the  request  of  the 
director,  Secretary  of  the  Navy’s  Office  of  Small  Business  Programs,  on  improving  DoN 
HUBZone  contracting.6  As  such,  it  is  necessary  to  explore  DoN’s  HUBZone  goaling 


1  The  HUBZone  Program  was  created  by  the  Small  Business  Reauthorization  Act  of  1997,  Public  Law 
105-135,  Title  VI,  codified  at  15  U.S.C.  §657a  (1997);  it  went  into  effect  in  March  1999  once  the  SBA 
established  certified  its  first  HUBZone  firm.  See  Dilger  (2013). 

2  The  reference  to  rescue  duty  is  an  attempt  at  analogy  to  the  seafarers’  rescue  duty.  Historically, 
when  at  sea,  admiralty  and  international  maritime  law  gives  the  U.S.  Navy  and  the  U.S.  Marine  Corps 
the  freedom  of  navigation  but  also  imposed  on  them  the  duty  of  rescue  those  in  distress  (also  known 
as  the  duty  to  render  assistance).  The  policy  rationale  for  this  duty  is  the  recognition  that  seafarers 
must  use  their  powers  for  good  without  merely  waiting  for  some  other  rescue  ships  to  arrive.  See 
generally,  Maltzman  and  Ehrenreich  (2015);  Commander’s  Handbook  (2007),  Ch.  3,  “Protection  of 
Persons  and  Property  at  Sea  and  Maritime  Law  Enforcement”;  Peltz  (2014). 

3  See  generally,  FAR  Parts  5  and  6  (2015). 

4  See  15  U.S.C.  §644  (2007). 

5  See  15  U.S.C.  §657a  (2015);  FAR  Subpart  19.13;  13  C.F.R.  Part  126  (2015). 

6  This  study  extensively  relies  on  the  analytical  framework  and  conceptual  analysis  from  a  prior 
SECNAV  OSBP-sponsored  study:  Kidalov  and  Lee  (2015) 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-252- 


performance  and  the  HUBZone  Program’s  background  to  place  the  study  in  its  proper 
context.  Data  in  Figure  1,  Figure  2,  and  Table  1  show  that  the  DoN’s  goaling  performance 
needs  a  turnaround. 
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Figure  1.  DoN  HUBZone  Goal  Achievements  Across  HUBZone  Program’s  History 
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Figure  2.  DoN  HUBZone  Goaling  Spending  Across  HUBZone  Program’s  History 
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Table  1. 


DoN  HUBZone  Historic  Goaling  Results;  Identification  of  the  Period 

Under  Study 


DON  HUBZone  Goaling  Results  across  HUBZone 
Program  history 
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0.32ft 
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At  the  time  of  the  HUBZone  Program’s  design  and  since,  its  design  has  been  the 
subject  of  intense  criticisms.  U.S.  Senate  Small  Business  and  Entrepreneurship  Committee 
Chairman  Senator  Kit  Bond  intended  for  the  HUBZone  to  replace  the  8(a)  Business 
Development  Program  for  Small  and  Disadvantaged  Businesses,  while  President  Clinton 
and  others  in  Congress  did  not  (e.g.,  see  Dilger,  2013).  Others  criticized  the  HUBZone 
program  as  interfering  with  efficiency,  or  as  favoring  “close  swap”  of  non-HUBZone  firms  for 
HUBZone  firms  that  were  close  to  winning  contracts  even  without  HUBZone  assistance 
(e.g.,  see  Dilger,  2013;  Reece,  2011). 

As  noted  in  the  author’s  prior  research  (Kidalov  &  Lee,  2015),  Section  8(a)  of  the 
Small  Business  Act  authorizes  and  directs  SBA  to  provide  business  development  assistance 
to  small  disadvantaged  businesses  (SDBs),  typically,  members  of  groups  victimized  by  past 
racial  discrimination.  Included  in  this  assistance  are  tailored  business  development  plans, 
pool  of  contract  requirements  “accepted  into”  the  program  for  not  more  than  seven  years  for 
sole  source  and  competitive  set-asides,  management  and  technical  advice,  agency  goals, 
training,  and  other  assistance.  In  addition,  8(a)  sole  source  contract  awards  can  be  made 
based  on  such  business  development  plan  even  if  there  is  another  willing,  but  more 
successful  8(a).  SBA’s  assistance  mix  would  change  as  the  firms  established  past 
performance  and  progressed  towards  program  graduation.  The  SBA  reports  to  Congress 
annually  on  assistance  metrics,  including  number  of  firms  assisted  and  agencies’  spending 
goal  achievement.7  Federal  and  DoD  contracting  officers  make  non-competitive  8(a)  awards 


7  See  generally  15  U.S.C.  §  637(a)  (2014);  13  CFR  Part  124  (2014);  SBA  Office  of  Business 
Development  (2008);  SBA  (2014). 
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with  SBA  direction  or  concurrence,  and  may  not  rededicate  8(a)  Program  contracts  for  other 
businesses  without  SBA  approval.  This  assures  the  8(a)  Program  a  “floor”  in  terms  of 
program  spending  and  breadth  of  industries.  Further,  the  8(a)  Program  increases  the 
outcome  of  business  development  of  disadvantaged  entrepreneurs  through  a  firm-focused 
process  there  the  SBA  assumes  much  of  the  responsibility  for  picking  firms  in  need  of 
contract  awards,  leaving  contracting  officers  to  focus  on  better  requirements  definition  and 
contract  administration.  In  contrast,  the  SBA  emphatically  stated  that  no  business 
development  assistance  will  be  provided  to  individual  HUBZone  firms,  but  that  assistance 
will  be  indirect  to  HUBZone  communities  at  large.8  In  essence,  DoN  contracting  officers  bear 
some  moral  or  public  interest  responsibility  for  HUBZone  economic  development,  but  well- 
defined  responsibility  for  performance  risk  and  no  specific  guidance  on  how  to  tailor 
contracts  to  HUBZone  firms’  needs. 

Since  the  creation  of  the  HUBZone  Program,  the  legal  force  of  this  economic 
assistance  duty  has  been  the  subject  of  an  intense  debate.  Between  1999  and  2005,  this 
duty  was  legally  mandatory  and  took  precedence  over  assistance  duties  to  other  types  of 
small  businesses.  In  August  2005,  however,  the  U.S.  Small  Business  Administration 
decided  that  the  duty  to  the  HUBZone  firms  must  be  subject  to  parity  with  other  so-called 
socioeconomic  small  business  categories.9  In  the  January  11, 2006,  case  of  Contract 
Management  Industries,  Inc.  v.  Rumsfeld, 10  which  concerned  HUBZone  set-aside  at  Naval 
Base  Pearl  Harbor,  the  DoN  took  a  very  firm  stand  against  the  SBA’s  position  and  in  favor  of 
the  original  mandatory  design  of  the  HUBZone  Program  set-asides  and  the  precedence  of 
HUBZone  set-asides  over  the  8(a)  Program.  The  U.S.  Court  of  Appeals  for  the  9th  Circuit, 
and  at  the  time,  the  U.S.  Department  of  Justice  sided  with  the  DoN  against  the  SBA.  In 
International  Program  Group,  /nc.,* 11  a  September  19,  2008,  case  involving  pre-deployment 
training  contracts  at  Marine  Corps  Camp  Pendleton,  the  DoN  doubled  down  on  that  victory. 
The  DoN  obtained  a  GAO  opinion  reinforcing  the  9th  Circuit  and  the  precedence  of 
HUBZone  set-asides  over  set-asides  for  service-disabled  veteran-owned  small  businesses 
(SDVOSBs).  However,  the  debate  did  not  stop.  The  U.S.  Government  Accountability  Office, 
the  U.S.  Court  of  Federal  Claims,  the  DoJ  Office  of  Legal  Counsel,  the  Executive  Office  of 
the  President,  Congress,  and  various  DoD  components  all  added  to  the  debate  (Branch, 
2009).  On  August  4,  2009,  the  DoN  acceded  to  the  view  of  the  SBA,  DoD  Defense 
Procurement  and  Acquisition  Policy,  and  DoJ  OLC  that  HUBZone  contracting  set-asides 
must  have  parity  with  other  Small  Business  Act  socioeconomic  categories  and  be  subject  to 
the  discretion  of  the  contracting  officer  (Branch,  2009).  The  debate,  which  lasted  through 
201 0-201 1 ,  has  triggered  a  constitutional  stand-off  over  congressional,  judicial,  and 
executive  powers  over  financial  assistance  to  distressed  areas  by  means  of  government 
procurement  contracts.  The  crisis  was  resolved  when  Congress  acceded  to  the  Executive 
Branch  requests  and  legislated  parity  between  socio-economic  category  set-asides  in  the 
Small  Business  Jobs  Act  of  2010. 12 


8  See  U.S.  Small  Business  Administration,  HUBZone  Empowerment  Contracting  Program,  Final  Rule, 
63  Fed.  Reg.  31896-31916  (June  11,  1996). 

9  See  International  Program  Group,  Inc.,  B-400278;  4-00308  (GAO)  (Sept.  19,  2008). 

10  See  Contract  Management  Industries,  Inc.  v.  Rumsfeld,  434  F.3rd  1145  (9th  Cir.  2006). 

11  See  International  Program  Group,  Inc.,  B-400278;  4-00308  (GAO  Sept.  19,  2008). 

12  Public  Law  111-240  (Sept.  24,  2010). 
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As  noted  in  the  author’s  prior  research  (Kidalov  &  Lee,  2015),  the  SBA  and  FAR 
Council  amended  their  respective  regulations  in  201 1  and  2012.  The  SBA  amended  its 
regulations  on  February  4,  201 1  and  made  market  research  for  purposes  of  considering 
HUBZone  set-asides  mandatory: 


After  conducting  market  research,  the  contracting  officer  shall  first  consider  a 
set-aside  or  sole  source  award  (if  the  sole  source  award  is  permitted  by 
statute  or  regulation)  under  the  8(a)  BD,  HUBZone,  SDVO  SBC  or  WOSB 
programs  before  setting  aside  the  requirement  as  a  small  business  set-aside. 
There  is  no  order  of  precedence  among  the  8(a)  BD,  HUBZone,  SDVO  SBC 
or  WOSB  programs.  The  contracting  officer  must  document  the  contract  file 
with  the  rationale  used  to  support  the  specific  set-aside,  including  the  type 
and  extent  of  market  research  conducted.  (13  C.F.R.  §  1 25. 1 9(b)(2)(i),  2011). 


The  FAR  amendments  provided  for  parity  and  also  expressly  mandated 
consideration  of  the  HUBZone  Program  and  other  socio-economic  programs  before 
proceeding  with  regular  small  business  set-asides  above  the  Simplified  Acquisition 
Threshold  (SAT).  The  FAR  Council  (2012)  states,  “FAR  19.203(d)  was  added  to  include 
language  consistent  with  13  CFR  125.2(f)(2)(ii)  regarding  the  minimum  elements  a 
contracting  officer  should  examine  when  choosing  a  socioeconomic  program:  The  results  of 
market  research  and  progress  in  fulfilling  agency  small  business  goals.”  Moreover,  the  SBA 
and  FAR  regulations  provide  for  discretionary  (“may”)  rather  than  “shall”  mandatory 
language.  Based  on  the  Small  Business  Jobs  Act  of  2010,  Public  Law  111-240  (Sept.  24, 
2010),  FAR  §  19.203  outlines  three  general  rules  of  precedence  for  open  market 
procurements.  First,  “[s]mall  business  set-asides  have  priority  over  acquisitions  using  full 
and  open  competition”  (FAR  19.203).  Second,  “there  is  no  order  of  precedence”  among  the 
four  small  business  socioeconomic  programs:  the  8(a)  Program,  the  HUBZone  Program,  the 
Service-Disabled  Veteran-Owned  Small  Business  (SDVOSB)  Procurement  Program,  or  the 
Women-Owned  Small  Business  (WOSB)  Program.  The  choice  among  the  socioeconomic 
programs  is  discretionary,  in  that  “the  contracting  officer  should  consider,  at  a  minimum — (1) 
results  of  market  research  that  was  done  to  determine  if  there  are  socioeconomic  firms 
capable  of  satisfying  the  agency's  requirement;  and  (2)  agency  progress  in  fulfilling  its  small 
business  goals”  (FAR  19.203).  The  third  rule  concerns  the  choice  between  small  business 
set-asides  and  small  business  socio-economic  set-asides.  However,  the  parity  between 
those  programs  is  still  subject  to  the  8(a)  claw-back  priority:  “However,  if  a  requirement  has 
been  accepted  by  the  SBA  under  the  8(a)  Program,  it  must  remain  in  the  8(a)  Program 
unless  the  SBA  agrees  to  its  release  in  accordance  with  13  CFR  parts  124,  125,  and  126” 
(FAR  19.203).  The  8(a)  Program  also  retained  the  so-called  non-advertisement  rule  in  Part 
124,  which  obligates  contracting  officers  not  to  advertise  8(a)  requirements  through  non-8(a) 
programs. 

Since  201 1-2012,  the  federal  HUBZone  Program  also  suffered  a  decertification 
crisis  due  to  U.S.  Census  redesignations  of  HUBZone  areas.  As  many  as  30%  of  HUBZone 
firms  were  decertified  (Lee,  2012). 

Based  on  the  HUBZone  Program’s  background,  this  study  addresses  the  following 
three  research  questions: 


1 .  Can  DoN  HUBZone  Program’s  struggles  be  better  explained  in  terms  of  the 
generally  accepted  Cohen-Eimicke  Contract  Management  Performance 
Model  (inputs,  process,  outputs,  and  outcome)? 

2.  Are  measures  such  as  broad  and  unguided  individual-level  contracting  officer 
discretion  on  set-asides,  parity  with  the  8(a)  and  other  socioeconomic 
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program  set-asides,  and  Simplified  Acquisitions  effective  to  support 
HUBZone  participation  in  Navy  and  Marine  Corps  contracting? 

3.  What  should  DoN  do  to  turn  around  its  HUBZone  Program? 

This  study’s  research  hypothesis  is  that  the  DoN  HUBZone  Program’s  design  is 
misaligned  from  critical  performance  management  criteria  that  do  not  include  goaling 
spending.  Ironically,  this  misalignment  operates  to  impede  HUBZone  goaling  achievements 
over  the  long  run.  The  study  uses  the  Cohen-Eimicke  model  to  define  effective  program 
management  of  a  socioeconomic  contracting  program,  and  then  examines  Federal 
Procurement  Data  System  data  corresponding  to  the  Cohen-Eimicke  criteria.  Finally,  the 
study  makes  recommendations  for  DoN  and  SBA  action. 

Theoretical  Foundations  of  Effective  Program  Design:  Applying  the  Cohen- 
Eimicke  Contract  Management  Performance  Model  to  Hubzone 
Socioeconomic  Contracting 

The  description  of  the  Cohen-Eimicke  model  generally  follows  the  description 
contained  in  the  author’s  prior  research  on  the  SDVOSB  Program  based  on  similar 
methodology  (Kidalov  &  Lee,  2015).  Cohen  and  Eimicke’s  2008  modern  classic  The 
Responsible  Contract  Manager,  sorts  contracting  programs’  performance  measurements 
according  to  four  types  of  measures:  input(s),  process(es),  output(s),  and  outcome(s).13 
Inputs  are  a  measurement  of  program  resources,  such  as  “dollars  appropriated  and 
allocated,  ...  length  of  time  committed  to  the  problem,”  involvement  of  other  organizations, 
and  so  forth  (Cohen  &  Eimicke,  2008). 

Input  measures  are  frequently  criticized  because  they  tell  you  only  how  hard 
you  are  trying  to  do  something  about  a  problem  or  the  extent  of  your 
commitment  to  reach  a  particular  goal.  ...  Input  measures  tell  you  very  little 
about  how  well  you  are  doing  in  reaching  the  objective — they  measure  effort 
much  better  than  they  assess  results.  But  input  measures  should  not  be 
ignored.  They  provide  an  important  barometer  of  the  scope  of  activity  and  of 
the  present  and  future  demand  on  overall  resources,  serve  as  surrogates  of 
the  organization’s  priorities,  and  often  reflect  the  organization’s  customer 
preferences  as  well.  (Cohen  &  Eimicke,  2008,  p.  152) 

In  the  HUBZone  Program,  performance  (inputs)  is  generally  measured  by  means  of 
the  statutory  3%  prime  contracting  goal  under  the  Small  Business  Act  (or  goals  as  may  be 
negotiated  by  the  SBAs)  which  provides  the  “floor”  spending  share  of  agency  contracts  that 
should  go  to  HUBZone  small  businesses  (Cohen  &  Eimicke,  2008,  p.  153). 

Process  is  the  second  performance  measurement  (i.e.,  step  or  steps  involved  in 
generating  outputs,  such  as  production  of  items);  it  is  described  by  Cohen  and  Eimicke  as  a 
function  of  total  quality  management  (TQM).  “Measurement  of  those  activities  facilitates 
organizational  learning  and  improvement.  Process  measures  include  the  delineation  and 
definition  of  specific  work  steps,  measures  of  the  amount  of  time  it  takes  to  perform  specific 


13  As  noted  in  the  author’s  prior  research,  this  book’s  reception  is  discussed  in  Girth  (2014),  Joaquin 
(2010),  and  Filipovitch  (2010).  This  book  is  also  used  in  the  Naval  Postgraduate  School  contract 
management  curricula. 
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tasks,  error  rates,  and  similar  indicators.  Requiring  organizational  units  to  report  process 
measures  can  signal  government’s  concern  for  the  quality  and  efficiency  of  an 
organization’s  internal  operations  and  can  compel  attention  to  these  fundamental 
management  issues”  (Cohen  &  Eimicke,  2008,  p.  153).  In  the  HUBZone  Program,  HUBZone 
set-asides  as  well  as  related  publicity  and  market  research  to  meet  the  set-aside  Rule  of 
Two14  or  to  find  a  HUBZone  sole  source  contractor  constitutes  Program  process.  On  the 
other  hand,  non-HUBZone  set-asides  or  unrestricted  contracts  awarded  without  the  benefit 
of  the  price  evaluation  preference  constitutes  process  outside  of  the  HUBZone  Program. 

Output  is  the  third  performance  measurement  category,  which 

seek[s]  to  quantify  the  amount  of  work  accomplished  with  the  inputs  or 
resources  provided.  Output  measures  can  seek  to  measure  quantity,  quality, 
or  both.  Typical  output  measures  include  customers  or  clients  served,  facility 
condition  and  cleanliness,  miles  of  road  paved,  ...  or  number  of  products 
sold.  ...  Utilizing  a  select  number  of  indicators  that  have  a  direct  impact  on 
performance  (particularly  for  customers  and  funding  agencies)  leads  to  a 
successful  performance  measurement  system.  (Cohen  &  Eimicke,  2008,  p. 
153-154) 

A  typical  output  measure  for  the  HUBZone  Program  would  be  the  number  of 
HUBZone  small  businesses  that  benefitted  from  the  HUBZone  program,  or  a  number  of 
contracts  awarded  through  the  HUBZone  Program  (Cohen  &  Eimicke,  2008). 

Outcome-  or  impact-based  measures  are  the  fourth  and  final  category.  They  assess 
whether  the  desired  objective  or  state  is  being  achieved.  As  Cohen  and  Eimicke 
acknowledged,  outcomes  are  difficult  to  define  and  measure.  In  general,  “the  function  of 
performance  management  remains  the  same:  What  are  we  trying  to  do,  and  are  we 
succeeding  in  doing  it?”  (Cohen  &  Eimicke,  2008,  p.  155).  For  the  HUBZone  Program, 
Section  606  of  Public  Law  “increased  employment  opportunities  and  an  increased  level  of 
investment  in  HUBZones”  and  further  defines  those  terms  by  reference  to  Section  602  to 
mean  “Federal  contracting  assistance”  provided  in  accordance  with  the  HUBZone  Program. 
The  former  part  of  this  convoluted  definition  could  include,  for  example,  factors  such  as  the 
diversity  of  industries  in  which  HUBZone  firms  participate  or  the  diversity  of  goods  or 
services  requirements  which  they  provide.  The  latter  part  of  the  definition  appears  to 
duplicate  inputs-based  measures. 

Understanding  the  Hubzone  Program  Operations  Through  the  Cohen-Eimicke 
Contract  Management  Performance  Model 

As  stated  above,  this  study  generally  follows  the  methodology  of  the  Kidalov-Lee 
(2015)  study  on  SDVOSB  contracting.  Thus,  the  methodological  explanations  and  data 
comparisons  in  this  section  follow  or  closely  parallel  the  Kidalov-Lee  study. 


14  The  Rule  of  Two  refers  to  a  contracting  officer’s  determination,  prior  to  a  set-aside,  that  two  or  more 
capable  HUBZone  firms  are  willing  to  submit  offers  at  fair  market  prices.  See  15  U.S.C.  §  657a 
(2015). 
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HUBZone  Program  Taxonomy:  Inputs — Overall  Trends  on  DoN  Spending  With 
HUBZone  SBCs 

To  understand  the  full  investment  value  of  DoN  HUBZone  contracting,  it  is 
necessary  to  examine  the  spending  data  attributable  not  only  to  HUBZone  goals  but 
also  to  net  spending  (through  new  awards  and  modifications)  as  well  as  through  new 
awards  alone.  Because  the  HUBZone  Program’s  legal  and  management  authorities 
concern  New  Awards  and  not  contract  modifications,  this  study’s  primary  focus  is  on  New 
Awards.  References  in  this  study  to  “New  Awards”  or  “Awards”  will  be  interchangeable. 

The  FPDS  Goaling  Report  spending  data  typically  contain  New  Awards  and  various 
accretive  modifications  such  as  options;  this  data  does  not  cover  deductive  modifications 
(such  as  terminations)  and  is  subject  to  goaling  exclusions  (such  as  overseas  contracts) 
(U.S.  General  Services  Administration  [GSA],  2015;  SBA,  2003;  Kidalov  &  Snider,  2013). 
This  accounts  for  varying  levels  between  goaled,  net,  and  New  Awards  data.  The  New 
Awards  data  show  the  value  of  all  DoN  contracts  with  HUBZone  firms  (not  to  be  limited  to 
“HUBZone  contracts”  through  Program  mechanisms)  identified  with  “Modification  0.”  The 
Net  Total  Spending  data  show  the  net  sum  of  all  DoN  HUBZone  contract  spending  with  all 
modifications  and  regardless  of  goaling  exclusions.  HUBZone  Program  contracts  are  set- 
aside  contracts,  which  provides  for  a  more  direct  comparison  to  other  programs  that  lack 
tools  as  the  price  evaluation  preference  (PEP).  PEPs  are  not  addressed  in  this  study  due  to 
data  quality  concerns.15  In  recognition  of  the  SBA’s  2005  position  favoring  parity  of 
socioeconomic  programs,  references  below  to  Parity  Programs  mean  the  8(a),  Women- 
Owned  Small  Business,  and  Service-Disabled-Veteran  Owned  Small  Business  programs  as 
appropriate  for  the  particular  year  at  issue. 

DoN  HUBZone  Spending  Trends 

Data  in  Figure  2,  Figure  3,  and  Table  2  below  help  understand  the  DoN’s  overall 
investment  in  HUBZone  firms  over  time,  as  well  as  the  component  contributions  to  that 
investment  from  the  HUBZone  Program,  parity  programs  and  other  non-program  award 
mechanisms.  That  relationship  between  overall  and  component  inputs  contributions  is 
important  for  the  Cohen-Eimicke  evaluation  framework. 

Evaluated  spending  categories  include  Goaling,  Net,  New  Awards,  HUBZone 
Program  (Set-Asides),  Non-HUBZone  Set-Asides  (covering  Parity  Programs  and  regular 
Small  Business  Set-Asides),  as  well  as  separate  data  for  8(a)  and  combined  Parity 
Programs.  DoN  HUBZone  spending  across  goaled  and  all  other  program  and  non-program 
categories  discussed  in  the  following  figure  and  table  below  has  peaked  in  FY2009.  While 
the  decline  of  spending  was  particularly  pronounced  in  FY2013  sequestration  year,  Net 
Losses  appear  to  be  peaking  post-sequestration. 


15  For  example,  in  FY2011  FPDS  data,  HUBZone  price  evaluation  preferences  as  high  as  60%  were 
recorded.  In  contrast,  15  U.S.C.  657a  statutory  price  evaluation  preference  is  generally  set  at  no 
more  than  10%. 
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DON  HUBZone  Program  and  Non-Program  Spending 
on  HUBZone  Prime  Contracts 

$  1,800,000,000.00 
$  l.ttfl.LXXMXtS.OO 
51,400,000,000.00 
51,200,000,000.00 
$1,000,000^000.00 
$eoo,ooaooo.oo 
SGW»Q ,00000 
$4M,J00O>M0.0O 
$200000,000.00 


>u.yu 

FY  fy  FY 

0*  07  06 

FV 

09 

FY 

10 

FY 

11 

FY  FV  FY  FY 

12  13  14  IS 

— ^DON  Coaling  Report  SjMiiding 

$897,61  $1,104.  51,337, 

S1750. 

$13W, 

$1,387, 

$  1,367, $S9B,1 5  5 1,109.  $693,14 

HUB  Zone  mi  Total  5pcnd«nn 

5894,73  51,121.  51.321, 

51,679, 

51,399, 

$1,344, 

5 1.443, 5936,4651,147.  $934*57 

— HUBiune  New  Awards  Spending 

5 745,095®  3,30  51,1 14, 

$1,586, 

$1,111, 

$1,067, 

5 1,036,  $7?S,64$96S,SI$746,1G 

- HUB  Zone  Set-Aside  New  Awards 

593,9705 143,635187,3^436, 745342, 995337,?0S27S, 2551 73,555303,955194,07 

— ^Non-HUBzone  5tt-Aside  Mew 
Awards 

$352,$£WML&e$«  U$W  l,$*$Sl 1,345552,9  7$3$8Hks2fcSf$4$8^ 

Figure  3.  DoN  Program  and  Non-Program  Spending  by  Reference  to  Goaling 

Report 


S  £ 
1  = 

ii 

■c  =1 


DON  HUBZone  Program  and  Non-Program  Spending 
on  HUBZone  Contracts 

si.ooo.oGo.ewojoa 
51^00,000,000.00 
51,200,000,000.00 
51,000,000,000,00 
saoo.ooaewo.oo 
5600,000,000,00 
5^00,000,000.00 
Sioo,oaaewo,<w 


$0.00 

■  S3  00^000,000 .  OQ 

FY  iY  FY 

FY  FY 

y  f  FY  FY  FY  FY 

06  07  08 

09  10 

11  12  IS  14  15 

=*^  HU63nriP  Mew  Awjrd*  Spending 

$M5,(KSW3,3y$l,114, 

$1,506,  51-113. 

$1,067,  $1JG36, 5  7Z8.fr4$965,51$ 746,10 

^^Nel  Revenue  losses 

-$7,290  -$14,81 

-$2,065  -$16,35 

-$6,578  $5,419 -$13,93  -$17,58  $2  3,63 

-  HUB  Zone  Sct-As-de  New  Awards  $93,670514  3,835  187,765436,74574  3,995337,705378 3 55 173,555 707,955194,03 
-Noil  HUBlMlf:  set  A*iefe  New AwJfdsSBS 7,955429, 5*5673,305776,1 7563 1,56551 1,3*555, 3,975355,205576,51541896 


-S^t  Set  Aside  New  Aw.hUs 


$3l  V$360£S$$3  1^6583  1$-U  /,04$36?,22$3Z  3,  dU$  166.4  /$23  Vl  BSl  W*W 


-  Parity  rrqjpjm?  $cl -Aside  New 
Award* 


$314,7053*1, 05553*435666,005418,3  75375,1 1&338.875 188,935775,03519818 


Figure  4.  DoN  HUBZone  Spending  Through  HUBZone  Program  and  Non-Program 

Contracting 


Wstantiaper  SC/£NTVaa77 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-260- 


Table  2.  DoN  HUBZone  Spending  Through  HUBZone  Program  and  Non-Program 

Contracting 


DON  HUBZone  5  pend  ingdiroL^h  HUEZune  Pribram  -and  Non- Program  Contra cts 


Fiscal 

Ytar 

D-U  N  Goa  lin.g  R  sp  crt 
Span  ding 

HUBZone'  Net 
Total  5  pending 

HUBZone  New 
Awa  rds^  $  pend  ir^; 

Net  Reve-nue 

Losjes 

l-UBZone  Set- 

Aside  New 

Awards 

Non-HUBzone 

5et-  Aside  New 

Awards 

S(a}Sel-Asde 
New  Awa  ills 

Pa  rity  Prog  ra  m  s 
Set-Aside  New 

Avoids 

FY06 

$597,613,942.79 

$745.09925654 

-$229096601 

$314,766,915.43 

$314,766.915 .43 

FY07 

$1,104,217,560.49 

$1121659.905.51 

$902305,159.52 

-$4*47*5324 

$143*3-3407. 35 

$42926262005 

$360,659,193.46 

$36105*467*5 

FY05 

$1,237, 221, 102.72 

$1221626,11721 

$1, 114.275,420. 30 

-$14,517345.51 

$157,76669003 

$623*00,796*0 

$531, 9-51,515 . 5  2 

$536,432,7692  2 

FY09 

$1,750,105,357.60 

$1,579,156,604.79 

$t  556369  272 12 

-$2,055,011.95 

$436,741575.96 

$775,12096033 

$659,313,495.91 

$666,664.33737 

FY10 

$1259,237,241.73 

$1,999,951,06492 

$1,111.42299*15 

-$16,554,77520 

$241991 5  7204 

$6212669*3.14 

$417,046, 206.09 

$415,271,395.75 

FY  11 

$1237,275,332.19 

$1344456,79191 

$1.067,764655.34 

-$6273,145*7 

$337,20946135 

$511,340*44*1 

$362,223,313.69 

$375,117*56.76 

FY12 

$1267, 520, 177.22 

$1443,154,13497 

$1036.44*  725. 20 

-$5419,725.15 

$27*25532411 

$552,976,979.76 

$323,304,399.22 

$335*73946.65 

FY  13 

$596, 154,375.94 

$9 33467,793 27 

■  •  il2.r3£."54.2s 

$17355635424 

$355202,443 .64 

FY  14 

$1109,412,329.02 

$ 1, 14 7.-5 64,59 2 .6 2 

$965,510745.11 

|  -$L7, 556*1727 

$20195105  135 

$52621321125 

$237, 1*5,933. 60j 

|  $275O37j092.11 

FY15 

$924979903.17 

$746106227.15 

|  $194024167.30 

$4 35965, 123 20 

$186,907, 135. 3o| 

Findings:  HUBZone  Program  spending  has  never  been  a  dominant  contributor 
to  DoN  HUBZone  contracting  investment  overall.  Following  the  2009  DoJ-  and  DoD- 
mandated  reversal  of  the  DoN  position  on  mandatory  precedence  of  HUBZone 
contracts  in  favor  of  contracting  officer’s  discretion,  DoN  contracting  officers  chose 
not  to  use  the  tools  they  previously  used  to  deliver  peak  spending  results.  In 
particular,  the  declines  in  HUBZone  Program  as  well  as  8(a)  and  other  Parity 
Programs  set-asides  appear  to  correlate  with  the  general  decrease  in  HUBZone 
spending  levels.  In  their  exercise  of  discretion  to  meet  spending  goals,  contracting 
officers  appeared  to  prefer  Non-HUBZone  set-asides,  particularly  regular  Small 
Business  Set-asides,  instead  of  HUBZone  set-asides.  The  continued  increase  in  Net 
Losses  shows  reluctance  of  contracting  officers  to  keep  work  with  HUBZone  firms. 

DoN  HUBZone  Program  Spending  Trends 

In  addition  to  overall  HUBZone  spending  and  Program  spending,  it  is  important  to 
examine  Program  spending  in  more  detail  by  type  of  set-aside  tool.  Data  in  Figure  5  and 
Table  3  show  HUBZone  Program  spending  over  the  years  by  competitive,  sole  source,  and 
SAP.  Just  as  the  data  above,  the  data  below  show  that  Program  spending  peaked  in 
FY2009.  Trends  below  suggest  that  HUBZone  Program  spending  is  unstable,  going  up  and 
down  from  year  to  year.  This  is  driven  primarily  by  volatility  in  competitive  set-asides 
spending.  HUBZone  sole  source  set-asides  spending  is  no  longer  a  serious  contributor  to 
Program  spending.  In  fact,  HUBZone  SAP  set-aside  spending  now  more  than  doubles 
HUBZone  sole  source  set-asides  spending. 
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DON  HUBZone  Program  (Set-Asides) 
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Figure  5.  DoN  HUBZone  Program  (Set-Asides)  New  Awards  Spending 
Table  3.  DoN  HUBZone  Program  (Set-Asides)  New  Awards  Spending 


DON  HUBZone  Program  Spending 

Fiscal 

Year 

HUBZone  Set- 

Asides  New 

Awards 

HUBZone  SAP 

Set-Asides 

New  Awards 

HUBZone 
Competitive  Set- 
Asrdes  New 

awards 

HUBZone  Sole 

Source  Set- 

Asides  New 

Awards 

FY0& 

51,577,755.00 

515,626,331.00 

FY  07 

$143,333,407.33 

$129,423,639.71 

514,404,717.67 

FY  OS 

5137,766,690.03 

5131,113,354.73 

56,652,335.25 

FY  09 

5436,741,575.96 

5411,350,236.32 

524,391,239.64 

FY  10 

5242,992,572.04 

55,940,175.07 

5226,336,262.40 

516,606,309.64 

FY  11 

5337,209,463.35 

55,231,539.77 

5310,009,272.52 

527,200,190.33 

FY  12 

5273,255,334.11 

52,035,736.53 

5262,516,053.37 

515,739,275.24 

FY  13 

5173,556,354.24 

51,413,996.22 

5172,140,360.66 

51,415,493.53 

FY  14 

5202,952,051.35 

53,711,114.10 

5202,553,233.69 

|  52,404,443.22| 

FY  15 

5194,024,167.30 

56,396,524.43 

5191,619,719.03 1 

Findings:  After  a  three-year  2007-2009  hiatus,  DoN  contracting  officers  rallied 
to  HUBZone  SAP  set-asides  in  FY2010  and  fully  restored  the  FY2012-2013  slump  in 
FY2015.  HUBZone  sole  source  set-asides  peaked  two  years  after  the  DoN  parity 
reversal,  but  took  a  substantial  drop  since.  DoN  spending  on  HUBZone  competitive 
set-asides  is  now  barely  half  it  was  in  FY2009.  It  appears  that  DoN  buyers  may  treat 
HUBZone  set-asides  as  a  risky  proposition  for  discretionary  spending,  except  for  low 
dollar  SAPs. 

HUBZone  Program  Taxonomy:  Process — Trends  on  Contracting  Officers’  Discretion 
to  Use  HUBZone  Set-Asides  and  Other  Contracting  Mechanisms 

In  the  Cohen-Eimicke  framework,  spending  input  trends  do  not  necessarily  explain 
the  contracting  officers’  use  of  the  contracting  process.  To  understand  process,  it  is 


PBAlSTANTlA  PER  SCIENTIAL 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-262 


necessary  to  examine  the  trends  in  the  HUBZone  Program  set-aside  actions  and  put  them 
in  context  of  other  non-set-aside  contract  actions. 

DoN  Contracting  Actions  With  HUBZone  SBCs 

Data  in  Figure  6  show  that  the  DoN  began  a  long  decline  in  New  Awards  to 
HUBZone  firms  as  early  as  2009,  after  peaking  in  FY2008.  This  decline  appears  to  have 
stabilized  over  the  last  three  fiscal  years.  Accretive  Modifications  (i.e.,  actions  to  direct 
funding  to  incumbent  HUBZone  contractors)  held  relatively  steady  through  FY201 1 ,  but  then 
slumped  off.  Competitive  HUBZone  set-aside  actions  peaked  in  FY2011,  while  the  sole 
source  set-asides  have  been  dropping  over  the  entire  decade  and  reached  anecdotal 
asterisk  levels.  Non-HUBZone,  Parity  Programs-only,  and  8(a)  set-asides  peaked  in 
FY2009. 
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DON  Contracting  Actions:  Spending  Tools 
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Figure  6.  DoN  Contracting  Actions:  Spending  Tools  for  HUBZone  Contracting 

Findings:  When  given  the  freedom  to  exercise  discretion  in  terms  of  set-aside 
program  choice,  DoN  contracting  officers  prefer  meeting  HUBZone  goals  through  the 
use  of  contracting  set-aside  tools  authorized  for  small  businesses  or  Parity 
categories  under  other  Small  Business  Act  programs.  This  cannot  be  simply 
attributed  to  the  loss  of  HUBZone  firms  through  decertification,  since  all  HUBZone 
awardees  under  other  set-aside  authorities  could  have  received  HUBZone  sole 
sources,  at  the  least.  Rather,  these  trends  appear  to  suggest  greater  comfort  with 
contracting  tools  under  other  programs. 

DoN  HUBZone  Sole  Source  Set-Aside  Awards  and  Their  Impact 

Data  in  Table  4  show  DoN  buyers’  use  of  HUBZone  sole  source  set-asides.  FY2006 
was  peak  year  for  such  actions  and  beneficiaries  of  such  awards.  The  spending  volume  of 
such  awards  peaked  in  FY201 1 ,  as  did  their  contributions  to  DoN  HUBZone  investment 
metrics.  However,  in  FY2014,  the  contributions  of  those  awards  hit  the  bottom  across 
actions,  beneficiaries,  and  spending. 
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Table  4.  DoD  New  HUBZone  Sole  Source  Set-Aside  Award  Trends;  Impact  on 

Market  Entry  and  HUBZone  Spending 


□  ON  New  HU  BZo  ne  Sole- So  u  rce  Awards:  Impact  on  Market  Entry  and 

Spending 

Fiscal 

Year 

HUBZone 

Sole 

Source 

Awards 

Share  of 

All  New 

HU  EZone 

Awards 

HUBZone 

Sole 

Source 

Awardees 

Share  of 

New 

HUBZone 

Awardees 

under  All 

Methods 

HUBZone  Sole 

Source  Set- 

Asides  New 

Awards 

Share  of  All 

New 

HUBZone 

Awards 

Spending 

Share  of 

Net  Total 

HU  EZone 
Spending 

Comparative 
Share  of 
-Goaling 
Repo  rt 

HUBZone 

Spending 

FY  OB 

fiZ 

Z.4K-:. 

52 

5.E7% 

$15,5Z5,fi31.00 

210% 

1.75% 

1.74% 

FY  07 

30 

0.94% 

ZB 

3.41% 

$14,404,717.57 

1.59% 

122% 

13  OS 

FYOS 

29 

O.BO% 

25 

3.00% 

$5,552,235.25 

0.50% 

0.50% 

0.54% 

FY  09 

ZO 

D.E1%. 

IE 

2Sf% 

5  Z4,B91,  ZB9.  E4 

1.57% 

1.32% 

1.4  2% 

FY  10 

IE 

0.50% 

10 

1.35% 

$15,505,309.54 

1.49% 

1.19% 

1.22% 

FY  11 

22 

0.71% 

12 

1.55% 

$27,200, 19D.B  3 

2.55% 

Z.02% 

211% 

FY  12 

19 

0.7B% 

10 

1.57% 

$  15, 7  39,  Z  75.  Z4 

1.52% 

1.09% 

1.15% 

FY  13 

_ 7 

0.5E% 

|  1.04% 

51,415,493.52 

_ 0.19% 

0.15% 

_ 0.15% 

F-i-  >1  'J  '  :  4 

FY  15 

e| 

1  0.41% 

E| 

|  1.5E% 

|  52,404,442.  Zz| 

0.32%J 

|  O.ZE% 

O.Z7%| 

Findings:  DoN  contracting  officers  clearly  disfavor  HUBZone  sole  source 
awards,  even  in  the  face  of  declining  HUBZone  goaling  spending  and  declining  base 
of  HUBZone  certified  firms.  The  contribution  of  this  contracting  method  to  HUBZone 
investment  or  market  entry  is  marginal.  Businesses  or  economic  development 
authorities  relying  on  its  availability  are  at  risk  of  disillusionment,  while  the  DoN 
potentially  misses  many  opportunities  for  increasing  its  HUBZone  spending  numbers. 

DoN  HUBZone  Competitive  Set-Aside  Awards  and  Their  Impact 

Data  in  Table  5  illustrate  the  use  and  role  of  DoN  competitive  HUBZone  set-asides. 
By  spending  volume,  these  awards  peaked  in  FY2009.  By  market  entry  contribution,  they 
peaked  in  FY2015,  but  not  because  of  a  peak  in  HUBZone  market  share  growth. 


Table  5.  DoN  New  HUBZone  Competitive  Set-Aside  Award  Trends;  Impact  on 

Market  Entry  and  HUBZone  Spending 


DON  New  HUBZone  Competitive  Set-Aside  Award  si  Impact  on  Market  Entry  and  Spending 

Fiscal 

Year 

HUBZone 

Co  mp  etitive 
Set-Aside 

Awards 

Share  of 

All  New 

HUBZone 

Awards 

HUBZone 
Competitive 
S  et-  As  id  e 

Awardees 

Share  of 

New 

HUBZone 

Awa  rd  ee  5 

under  All 

Methods 

HUBZone 
Competitive  Set- 
Asides  New 

awards 

Share  of 

All  New 

HUBZone 

Awards 

5  p  en  di  ng 

Share  of 

Net  Total 

HUBZone 

Spending 

C  om  pa  rative 

Share  of 

Go  al  i  ng 
Report 
HUBZone 

Spending 

FY06 

FT  07 

145 

4.57% 

93 

11.34% 

$129,428,689.71 

14.33% 

11.54% 

11.72% 

FYOS 

203 

5.57% 

77 

9.23% 

$131,113,554.73 

16.25% 

13.70% 

14.64% 

FT  09 

195 

5.94% 

105 

13.54% 

$411,350,236.32 

25.96% 

21.92% 

23.53% 

FT  10 

255 

3.31% 

121 

16.33% 

$226,386,262.40 

20.37% 

16.13% 

16.66% 

FT  11 

320 

10.39% 

132 

13.16% 

$310,009,272.52 

29.03% 

23.06% 

24.07% 

FT  12 

315 

13.06% 

106 

17.73% 

$262,516,053.37 

25.33% 

13.19% 

19.20% 

FT  13 

220 

11.31% 

35 

17.63% 

$172,140,860.66 

23.62% 

13.34% 

19.17% 

FT  14 

217 

10.39% 

92 

13.62% 

$202,553,233.69 

20.93% 

17.65% 

13.26% 

FT  15 

203 

10.69% 

91 

19.03% 

$191,619,719.03 

25.63% 

20.73% 

21.45% 

Findings:  DoN  contracting  officers  favor  HUBZone  competitive  set-aside 
awards  much  more  than  they  do  the  sole  source  set-asides. 
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DoN  Combined  HUBZone  Set-Aside  Awards  and  Their  Impact 

Data  in  Table  6  show  the  use  and  impact  of  combined  HUBZone  set-asides. 


Table  6.  DoN  New  HUBZone  Set-Aside  Award  Trends;  Impact  on  Market  Entry 

and  HUBZone  Spending 


DON  New  Combined  HUBZone  Set-Aside  Awards:  Impact  on  Market  Entry  and  Spending 

Fiscal 

Year 

HUBZone  Set- 

Aside  Awards 

Share  of 

All  New 

HUBZone 

Awards 

HUBZone 

S  et- As  id  e 

Awardees 

Share  of 

New 

HUBZone 

Awa  rd  ee  5 

under  All 

Methods 

HUBZone  Set- 

Asides  New 

Awards 

Share  of 

All  New 

HUBZone 

Awards 

S  p  en  di  ng 

Share  of 

Net  Total 

HUBZone 

Spending 

Comparative 

Share  of 

Goaling 

Report 

HUBZone 

Spending 

FY06 

175 

6.93  ft 

103 

12.19ft 

FY07 

117 

14.27ft 

1  5143,333,407.3s 

15.92ft 

12.32ft 

13.03ft 

FYOS 

224 

6.15ft 

100 

$137,766,690.03 

16.34ft 

14.21ft 

15.15ft 

FY09 

216 

6.55  ft 

121 

15.71ft 

$436,741,575.96 

27.53ft 

23.24ft 

24.96ft 

FY  10 

252 

3.31ft 

123 

17.27ft 

$24Z,99Z,57Z.04 

21.56ft 

1736ft 

17.53ft 

FY  11 

342 

11.10ft 

142 

19.53  ft 

$337,209,463.35 

31.53ft 

25.03ft 

26.13ft 

FY  12 

337 

1 3.35  ft 

115 

19.23  ft 

$273,255,334.11 

26.35ft 

19.23ft 

20.35ft 

FY  13 

227 

11.66ft 

13.67ft 

$173,556,354.24 

23.52ft 

13.49ft 

19.32ft 

FY  14 

222 

11.14ft 

97 

19.64ft 

5Z0Z,95Z,O51,35 

21.02ft 

17.69ft 

13.29ft 

FY  15 

216 

11.11ft 

99 

20.75  ft 

$194,024,167.30 

26.00ft 

20.99ft 

21.72ft 

Findings:  DoN  HUBZone  set-asides’  utilization,  and  impact,  are  fluctuating  and 
limited. 

DoN  HUBZone  Simplified  Acquisition  (SAP)  Awards  and  Their  Impact 

Data  in  Tables  7  and  8  suggest  that  FAR  Part  13  Simplified  Acquisitions  are  having  a 
positive  market  entry  impact.  However,  SAP  HUBZone  set-asides  appear  to  have  limited 
impact. 

Table  7.  DoN  SAP  HUBZone  Awards  and  Their  Impact  on  HUBZone  Market  Entry 

and  Contract  Spending 


DON  SAP  HUBZone  Awards:  Impact  on  Market  Entry  and  Spending 

Fiscal 

Year 

HUBZone 

SAP  Awards 

Share  of 

All  New 

HUBZone 

Awards 

HUBZone 

SAP 

Awardees 

Share  of 

New 

HUBZone 

Awardees 

underAII 

Methods 

HUBZoneSAP 

New  Awards 

Share  of 

All  New 

HUBZone 

Awards 

Spending 

Share  of 

NetTotal 

HUBZone 

Spending 

Comparative 

Share  of 
Goaling 
Report 

HUBZone 

Spending 

FY06 

1106 

43.77ft 

426 

43.0Sft 

$33,601,690.00 

4.51ft 

3.76ft 

3.74ft 

FY07 

21 

0.66ft 

9 

1.10ft 

FYOfl 

16 

0.44ft 

7 

0.34ft| 

1  $26,707,423.71 

2.40ft 

2.02ft 

216ft 

FY09 

$21,028, 346. 23 

1.33  ft 

1.12ft 

1.20ft 

FY  10 

660 

20.63ft 

202 

27.26ft 

$49,395,163.62 

4.44ft 

3.53  ft 

3.63  ft 

FY  11 

1153 

37.42ft 

293 

40.99ft 

$65,100,133.57 

6.10ft 

4.34ft 

5.05  ft 

FY  12 

634 

26.05ft 

225 

37.63  ft 

$32,061, 480. 52 

3,10ft 

2.12ft 

2.35  ft 

FY  13 

544 

27.95ft 

132 

37.76ft 

$21,011,097.89 

2.33ft 

2.24ft 

2.34ft 

FY  14 

743 

37.55  ft 

219 

44.33ft 

$44,259,672.95 

4.53ft 

3,36ft 

3,99ft 

FY  15 

797 

40.93ft 

231 

43.43  ft 

$43,697,377.56 

5.36ft 

4.73ft 

4.39ft 
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Table  8.  DoN  SAP  HUBZone  Set-Aside  Awards  and  Their  Impact  on  HUBZone 

Market  Entry  and  Contract  Spending 


DON  SAP  HUBZone  Set-Aside  Awards:  Impact  on  Market  Entry  and  Spending 


Fiscal 

Year 


HUBZore 
SAP  Set- 
Aside 
Awards 


Share  of 
AD  New 
HUBZone 
Awards 


HUBZone 
SAP  Set- 
Aside 
Awardees 


Share  of 
New 
HUBZone 
Awardees 
underAII 
Methods 


HUBZoneSAP 
Set-Asides  New 
Awards 


Share  of 
AH  New 
HUBZone 
Awards 
Spending 


Share  of 
Net  Total 
HUBZone 
Spending 


Comparative 
Share  of 
Coaling 
Report 
HUBZone 
Spending 


3.3996 


$1,577/755.00 


0.  n% 


am 


0.18% 


FY 10 

29 

0.91% 

18 

978% 

$5,340,175.07 

0.5396 

04296 

0.4496 

FY 11 

56 

1.81% 

46 

639% 

$5,231,530.77 

0.4996 

03996 

0.4196 

FY  12 

31 

1,27% 

24 

4.0196 

$2,085,736,53 

0.2096 

01496 

0.1596 

FY  13 

15 

0.77% 

14 

2.909b 

$1,413,336,22 

0.1996 

01,596 

0.1696 

FY  14 

11 

i.m% 

21 

4.2596 

$3,711,114.10 

0.3896 

03296 

0.3396 

FY  15 

34 

1,7596 

26 

5.4596 

$6,336,524,48 

0.8696 

0  6996 

0.7296 

Findings:  DoN  is  missing  the  opportunity  to  convert  HUBZone  SAP  awards 
into  HUBZone  set-asides. 

DoN  8(a)  and  Other  Non-HUBZone  Set-Asides  Awards  and  Their  Impact 

Table  9  and  Table  10  indicate  the  substantial  and  dominating  impact  of  non- 
HUBZone  set-asides  on  DoN  HUBZone  contracting. 

Table  9.  DoN  8(a)  Set-Aside  Awards  to  HUBZone  SBCs  and  Their  Impact  on 
HUBZone  SBCs  Market  Entry  and  Contract  Spending 


DON  8(a)  HUBZone  Set-Aside  Awards:  Impact  on  Market  Entry  a  nd  Spending 

Fiscal 

Year 

8(a)  Set- 

Aside  New 

Awards 

Share  of 

All  New 

HUBZone 

Awards 

8(a)  Set- 
Aside  New 

HUBZone 

Awa  rdee  s 

Share  of 

New 

HUBZone 

Awardees 

under  All 

Methods 

5(a )  Set-Aside 

N  ew  Awards 

Share  Df 

All  New 

HUBZone 

Awards 

Spending 

Share  of 

Net  Total 

HUBZone 

Spending 

Com  pa  ratrve 

Share  of 

Scaling 

Report 

HUBZone 

Spending 

FY  06 

638 

25.2596 

218 

24.6096 

$314,766,915.43 

42.2496 

35.2096 

35.0796 

FY  07 

748 

23.4396 

232 

25.2996 

$  3  60j  659, 19  3 .46 

39.9396 

32.1596 

32.6696 

FY  08 

881 

24. 1796 

241 

25.9096 

$531,951,515.52 

47.7296 

40.2596 

42.9996 

FY  09 

905 

27.4296 

216 

25.0596 

$659,313,495.91 

41.5696 

35.0996 

37.6796 

FY  lO 

764 

23.8896 

210 

25.3496 

$417,048,208.09 

37.5296 

29.8096 

30.6896 

FY  11 

673 

21.5496 

181 

24.9096 

$362,223,313.69 

33.9296 

26.9496 

28.1396 

FY  12 

543 

22.3196 

157 

26.2596 

$323,304,399.27 

31.1996 

22.4096 

23.6496 

FY  13 

379 

19.4896 

132 

FY  14 

_ 391 

19. 6396 

121  $237, 138,33  3,60 

24.5796 

20.6796 

21,3596 

FY  15  24. 9596 1  $186,907,138.30 

25.0596 

20.2296 

20.9396 
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Table  10.  DoN  Non-HUBZone  Set-Aside  Awards  to  HUBZone  SBCs  and  Their 
Impact  on  HUBZone  SBCs  Market  Entry  and  Contract  Spending 


DON  Non-HU  BZona  5  Et- As  ids  Awards:  Impact  do  Mark  at  Entry  and  Spandirg 


Fiscal 

Yaar 

Non- 

HUBZdue Set 

As  ids  New 

Awards 

5  ha  re  of 

All  New 

HUBZonE 

Awards 

Non- 

HUBZcte 

5Et-AsidE 

New 

HUBZdoe 

Awards  es 

S  h  a  re  of 

New 

HU BZdoe 

AwardEE  s 

under  All 

M  Eth  d  d  5 

Non-HUBzona 

5 Et-AsidE  New 

Awards 

Share  of 

All  New 

HUBZonE 

Awards 

Spanding 

Share  of 

Nat  Total 

HU BZdoe 

Spanding 

Comparative 

Sha  re  of 

Go  a  ling 

Re  port 

HUBZone 

Span  ding 

FY  06 

423 

39.47ft 

39.32ft 

FY  07 

134b 

42.1 5ft 

473 

57.63ft 

$429,562,620.03 

47.65  ft 

33.30ft 

FY  03 

1567 

42.99ft 

494 

59.23  ft 

$G  23,3lM>  j  79S.30 

5  6.9  5  ft 

47.20ft 

50.42ft 

FY  09 

1663 

47.62ft 

464 

60.26ft 

$773,120,360.33 

49.06ft 

41.41ft 

44.46ft 

FY  10 

1516 

47.33ft 

441 

59.51ft 

$621,566,333.14 

55.93ft 

44.42ft 

45.73ft 

FY  11 

1451 

47.1  Oft 

422 

53. 05ft 

$611,340,344.31 

47.39ft 

33.0  3ft 

39.70ft 

FY  12 

1162 

47.74ft 

360 

60.20ft 

S5  52,976,979.76 

53.35ft 

33.3  2ft 

40.43ft 

FY  13 

1021 

52.47ft 

316 

65.56ft 

$355,202,443.64 

43.7  6ft 

39.5  5  ft 

FY  14 

1144 

57.43  ft 

319 

64.57ft 

$5  26,513,611.55 

54.53ft 

45.33ft 

47.46ft 

FY  15 

1109 

57.02ft 

1  65.62ft 

$433,963,123.20 

53.33ft 

47.43ft 

49.15ft 

Findings:  DoN  HUBZone  program  has  a  substantial  dependency  on  non- 
HUBZone  set-asides. 

HUBZone  Program  Taxonomy:  Outputs — Trends  on  HUBZone  Participation  in  DoN 
Contracting 

To  appreciate  the  breadth  or  shallowness  of  DoN’s  HUBZone  contractors’  bench,  it  is 
important  to  examine  HUBZone  participation  in  both  Program  and  non-Program  DoN 
contracting. 

HUBZone  Participation  in  DoN  Contracting  Overall 

Data  in  Figure  7  show  HUBZone  participation  in  DoN  Contracting  and  across  various 
contracting  mechanisms.  Data  show  that  more  HUBZone  firms  participate  in  8(a)  set-asides 
and  non-HUBZone  set-asides  than  in  HUBZone  set-asides. 
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HUBZone  Participation  Trends  in  DON  Contracting 
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Figure  7.  HUBZone  Participation  Trends  in  DoN  Contracting  Overall 

Findings:  The  DoN  is  experiencing  a  crisis  of  HUBZone  contractor 
participation  in  DoN  contracting.  However,  the  DoN  is  not  using  HUBZone  set-asides 
to  reverse  this  crisis. 

HUBZone  Program  Participation  at  DoN. 

Data  in  Figure  8  illustrate  the  HUBZone  Program  participation  trends  as  a 
consequence  of  contracting  officer’s  discretion  to  set  aside  or  not  set  aside  work  for 
HUBZone  SBCs  on  a  competitive  or  sole  source  basis.  Overall,  this  data  show  that  DoN 
contracting  officers  are  not  exercising  discretion  to  increase  the  total  count  of  HUBZone 
firms  in  the  HUBZone  Program. 
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Participation  in  DON  HUBZone  Program 
Assistance: 

Trends  in  Contracting  Officers'  Discretion 
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Figure  8.  DoN  HUBZone  Program  Assistance  Participation:  Trends  in  Contracting 

Officers’  Discretion 

Findings:  There  is  a  crisis  of  participation  in  DoN  HUBZone  Program.  The 
HUBZone  Program  appears  to  have  limited  impact  as  the  entryway  into  the  DoN 
market.  The  DoN  should  act  to  reverse  these  trends. 

HUBZone  Program  Taxonomy:  Outcomes — Trends  Related  to  HUBZone  Program’s 
Industrial  Base  Diversity  and  DoN  Requirements  Matching 

To  evaluate  whether  HUBZone  contracting  creates  meaningful  jobs  and  a  diverse 
industrial  base  for  the  DoN,  it  is  important  to  evaluate  the  trends  in  codes  used  for 
identifying  industries  and  requirements.  North  American  Industrial  Classification  (NAICS) 
codes  are  assigned  to  each  contract  solicitation  for  use  in  market  research  across  industries 
as  well  as  determinations  whether  a  firm  is  small  by  reference  to  NAICS-based  business 
size  standards.  Other  codes,  the  Product  Service  Codes/Federal  Supply  Codes 
(PSCs/FSCs),  are  used  to  identify  what  is  actually  bought  (see  generally  Bunting,  2013). 

DoN  HUBZone  Program’s  Industrial  Base 

Figure  9  shows  NAICS  trends  across  various  contracting  mechanisms.  The  trends 
show  the  broadest  HUBZone  industrial  base  existed  in  FY2008. 
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NAICS  Trends:  Preserving  the  Industrial  Base 
lor  DON  HUBZone  Contracting 


3 

o 

■c 

£ 

5 


*  O 

fK 

Oft 

Ft 

07 

FY 

Oft 

FY 

SB 

FY 

III 

FY 

11 

ft 

12 

FY 

13 

fY 

14 

FY 

15 

-All  HUBZ4MW  NtwAwJrft 

146 

JWj 

193 

SSS 

362 

x:-i 

293 

274 

273 

-IHUBZorw  ift  toidc  Awvtidi 

59 

65 

ft* 

ft* 

70 

5* 

47 

SO 

-»£HV  1 1 : i:  ■/.  i.  iv  SCI- ASU 6 

Awirth 

ITS 

?43 

i&s 

231 

J47 

J4S 

iUl 

19S 

ZDS 

S*  t-Aiidei  U-ivU  Of 

IcSrnCif  k'-ct 

dOth 

74a 

301 

jyn 

JIjG 

JDl 

iHi 

1  l>H 

-Parity  Sit-Asll^ 

AwJrtfc 

ft? 

]m 

SOD 

lift 

■34 

3S 

71 

$2 

[iH 

J] 

-8<a]  HUBZone 

fimwAk 

« 

3d? 

9ft 

30? 

BO 

fA 

5B 

?0 

HUBZone  SAP  Awards 

2Sft 

5 

5 

4 

2il 

J?9 

217 

173 

'  2  IS 

70S 

-5APHUH/of>r  SeE- Aside 
AwahH 

2H 

0 

0 

0 

25 

■ID 

26 

13 

19 

21 

■SAP  1^A\  5ft-Awlt  Awgrdv 

45 

O 

Q 

n 

40 

SB 

U 

7B 

HI 

33 

Figure  9.  DoN  HUBZone  NAICS  Trends:  Use  of  HUBZone  Program  and  SAP  for 
Industrial  Targeting  and  Business  Development 

Findings:  The  number  of  industries  used  for  HUBZone  and  Non-HUBZone  set- 
asides  as  well  as  SAP  awards  have  recently  declined,  while  HUBZone  industries 
overall  began  falling  in  FY2008. 

HUBZone  Contractors’  Matching  to  DoN  Requirements 

Figure  10  shows  trends  in  matching  HUBZone  firms  to  DoN  requirements  as 
determined  by  PSCs/FSCs.  The  matching  trends  show  a  progressive  decrease  in 
PSCs/FSCs  satisfied  through  contracts  to  HUBZone  firms,  with  recent  increases  in  Non- 
HUBZone  set-asides  and  SAP  awards. 
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PSC/FSC  Trends:  Matching  HUBZone  Firms  to 
DON  Contracting  Requirements 
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Figure  10.  DoN  HUBZone  PSC/FSC  Trends:  Use  of  FlUBZone  Program  and  SAP  to 

Match  HUBZone  SBCs  to  DoN  Requirements 

Findings:  DoN  contracting  officers  consistently  prefer  to  fill  more  PSCs/FSCs 
through  8(a)  set-asides  and  other  non-HUBZone  set-asides. 

Answers  to  Research  Questions;  Recommendations  for  Hubzone  Program 
Reforms  and  Further  Research 

There  is  no  question  that  the  HUBZone  Program  has  suffered  from  serious 
adversities,  including  mass  decertification  and  the  parity  crisis,  as  well  as  the  effects  of 
sequestration.  Nonetheless,  through  the  Cohen-Eimicke  framework,  this  study  succeeds  in 
asking  and  answering  questions  about  how  the  DoN  can  more  effectively  manage  the 
HUBZone  Program  back  on  track  towards  success.  Concerning  the  first  question, 
whether  the  Cohen-Eimicke  Contract  Management  Performance  model  can  explain 
DoN  HUBZone  Program’s  performance  trends,  the  answer  is  “yes.”  The  focus 
predominantly  or  only  on  spending  goals  treats  HUBZone  Program  spending  inputs  as 
outputs.  DoN  contracting  officers  then  default  to  choosing  other  program’s  contracting  tools, 
whether  because  of  SBA’s  pronouncements  about  limited  assistance  to  HUBZone  firms, 
because  of  8(a)  requirements  retention  and  non-advertising  mandates,  or  because  non- 
HUBZone  set-asides  offer  the  best  possibility  to  reconcile  the  complex  contracting 
preferences.  Yet  confusing  this  fundamental  distinction  between  performance  criteria 
prevents  aligning  HUBZone  Program  process  tools,  such  as  set-asides,  to  true  outputs 
(number  of  participating  HUBZone  firms)  and  outcomes  (growth  in  HUBZone  industries  and 
capabilities).  When  the  DoN  begins  to  make  a  strategic  effort  to  target  HUBZone  set-asides 
towards  greater  number  of  HUBZone  firms,  the  HUBZone  goaling  spending  will  start 
increasing,  too. 


mtSTANTIA  PER  SCIENTIAL 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-271  - 


As  to  the  second  question,  whether  HUBZone  Program’s  parity  coupled  with 
unguided  discretion  of  contracting  officers  on  when  to  use  this  parity  to  target 
particular  HUBZone  firms  is  an  effective  approach  to  HUBZone  contracting,  the 
answer  is  “no.”  Overall,  DoN  buyers  appear  to  prefer  “close  swap”  transactions  where 
HUBZone  firms  get  the  work  because  they  are  already  established  under  other  Small 
Business  Act  programs.  Lacking  business  development  skills  and  SBA  business 
development  support,  DoN  contracting  officers  are  shying  away  from  HUBZone  set-asides 
even  to  already-successful  HUBZone  firms  that  obtain  DoN  contracts  through  other  tools. 
Moreover,  true  regulatory  parity  appears  to  be  lacking  with  the  8(a)  Program.  It  is  not 
surprising  that  8(a)  firms  by  and  large  maintained  their  share  of  the  DoN  HUBZone  Program, 
and  why  8(a)  firms  account  for  virtually  all  of  combined  Parity  Programs’  metrics  in 
HUBZone  contracting.  New  SAP  Awards  appear  to  be  playing  an  increasingly  favorable 
and  critical  market  entry  support  role  for  HUBZone  firms  seeking  DoN  contracts.  This 
role  should  be  strengthened. 

With  this  in  mind,  the  answer  to  the  question  as  to  what  DoN  should  do 
becomes  obvious.  Rebuilding  HUBZone  set-aside  participants’  bench  should  be  the  DoN’s 
first  step  at  increasing  the  HUBZone  Program’s  stability  and  then  turning  the  HUBZone 
Program  performance  around.  To  do  so,  the  HUBZone  Program  should  be  reformed  into  a 
business  development  program  similar  to  the  8(a)  Program  which  would  relieve  DoN 
contracting  officers  of  business  development  burdens.  The  program  will  be  focused  on  SAP 
and  set-asides.  Absent  SBA  initiative  in  that  regards,  the  DoN  should  craft  such  a  program 
for  itself  by  using  FAR  Part  6  and  the  Small  Business  Act’s  15  U.S.C.  §  644(a)  industrial 
base  support  authorities.  Finally,  further  research  on  HUBZone  contracting  topics  is 
recommended,  including  a  more  detailed  matching  of  contracting  trends  to  legal  and  policy 
authorities. 
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Panel  8.  Data  Policies,  Procedures,  &  Access: 
Illuminating  How  Acquisition  Information  Moves 
Within  the  Department  to  Support  Analysis  & 
Decision  Making 


Wednesday,  May  4,  2016 

3:30  p.m.  - 
5:00  p.m. 

Chair:  Mark  Krzysko,  Deputy  Director,  Acquisition  Resources  and  Analysis, 

OUSD  (AT&L) 

Discussant:  Ralph  DiCicco,  Acquisition  Chief  Information  Officer  (CIO),  United 
States  Air  Force 

Issues  With  Access  to  Acquisition  Data  &  Information  in  the  Department  of 
Defense:  Policy  &  Practice 

Megan  McKernan,  Defense  Research  Analyst,  RAND 

Jessie  Riposo,  Senior  Operations  Researcher,  RAND 

Issues  With  Access  to  Acquisition  Data  &  Information  in  the  Department  of 
Defense:  Doing  Data  Right  in  Weapon  System  Acquisition 

Nancy  Moore,  Senior  Management  Scientist,  RAND 

Megan  McKernan,  Defense  Research  Analyst,  RAND 

Chair: 

Mark  Krzysko — is  the  Deputy  Director,  Enterprise  Information,  Office  of  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  &  Logistics,  Department  of  Defense  (DoD).  He  directs 
acquisition  data  management  and  analysis,  technical  transformation,  and  shared  services  efforts  to 
make  timely,  authoritative  acquisition  information  available  to  support  oversight  of  the  DoD’s  major 
programs — a  portfolio  totaling  more  than  $1 .6  trillion  of  investment  funds  over  the  life  cycle  of  the 
programs.  Preceding  his  current  position,  Krzysko  served  as  Assistant  Deputy  Under  Secretary  of 
Defense  (ADUSD)  for  Business  Transformation,  providing  strategic  leadership  for  reengineering  the 
Department’s  business  improvement  decision-making  processes  and  leading  strategic  sourcing  and 
acquisition  process  efforts.  In  the  Office  of  the  Secretary  of  Defense,  he  also  served  as  Director  of  the 
Supply  Chain  Systems  Transformation,  championing  innovative  uses  of  information  technologies  to 
improve  and  streamline  the  supply  chain  process  for  the  Department.  Prior  to  that,  Krzysko  led  the 
Defense  Procurement  &  Acquisition  Policy  office  as  Deputy  Director  of  e-Business,  where  he 
provided  oversight  and  led  transformation  of  the  acquisition  community  into  a  strategic  business 
enterprise.  Prior  to  that,  Krzysko  led  the  Electronic  Commerce  Solutions  Directorate  for  the  Naval  Air 
Systems  Command  and  served  in  senior-level  acquisition  positions  at  the  Naval  Air  Systems 
Command,  including  Contracting  Officer  of  F/A-18  Foreign  Military  Sales,  F/A-18  Developmental 
Programs,  and  the  F-14. 

Discussant: 

Ralph  DiCicco — is  Deputy  Acquisition  Chief  Information  Officer  (CIO),  working  for  the  Deputy 
Assistant  Secretary  of  the  Air  Force  (Acquisition  Integration),  SAF/AQX.  The  Acquisition  CIO  is 
responsible  for  developing  architecture,  data  strategy,  systems  migration  plan,  and  a  portfolio 
management  process  for  the  acquisition  domain  (defined  as  one  of  14  domains  in  the  Air  Force 
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enterprise  architecture).  DiCicco  spent  25  years  on  active  duty  in  the  Air  Force  and  retired  as  a 
colonel  in  April  2006.  As  an  Air  Force  officer,  he  served  as  an  acquisition  program  manager,  test 
manager,  and  in  various  staff  positions  involved  in  planning,  resource  allocation,  and  congressional 
appropriations  liaison.  He  joined  federal  service  as  an  Air  Force  civilian  in  2006  and  served  as  Acting 
Director  of  the  Air  Force  Acquisition  Center  of  Excellence  as  well  as  Program  Manager  of  a  Major 
Automated  Information  Systems  program.  DiCicco  received  his  Bachelor  of  Science  degree  in 
mechanical  engineering  from  the  University  of  Massachusetts  Lowell  in  1980  and  a  Master  of  Science 
degree  in  electrical  engineering  from  George  Washington  University  in  1988.  He  is  a  graduate  of  the 
Defense  Systems  Management  College  Program  Managers  Course  and  Executive  Program 
Management  Course.  He  has  attended  executive  management  seminars  at  University  of  North 
Carolina  at  Chapel  Hill  and  the  Center  for  Creative  Leadership. 
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Issues  With  Access  to  Acquisition  Data  &  Information  in 
the  Department  of  Defense:  Policy  &  Practice 


Megan  McKernan — is  a  Defense  Research  Analyst  at  RAND.  McKernan  has  more  than  10  years  of 
experience  conducting  DoD  acquisition  analyses.  She  is  currently  co-leading  research  examining 
acquisition  data  sharing  in  the  DoD.  McKernan  has  also  conducted  analyses  on  other  defense 
acquisition  topics:  tailoring  the  acquisition  process,  program  manager  tenure,  and  root  causes  of 
Nunn-McCurdy  unit  cost  breaches.  She  uses  a  variety  of  methods  in  conducting  research,  including 
case  studies,  interviews,  and  literature  reviews.  She  holds  an  MA  in  international  trade  and 
investment  policy  from  George  Washington  University  and  a  BA  in  economics  from  William  Smith 
College,  [mckernan@rand.org] 

Jessie  Riposo — is  a  Senior  Operations  Researcher  at  RAND,  with  over  a  decade  of  experience  in 
research  and  analysis  with  a  specialty  in  Defense  Acquisition  and  Planning,  Programming,  and 
Budgeting.  Since  2003,  Riposo  has  led  and  participated  in  projects  in  support  of  the  USD(AT&L),  U.S. 
Navy,  UK  MoD,  and  the  Australian  DoD.  Riposo’s  projects  have  covered  reviews  of  domestic  and 
foreign  acquisition  programs,  Department  acquisition  policy  and  industrial  base,  personnel,  and  policy 
assessments.  Riposo  has  applied  a  variety  of  quantitative  and  qualitative  tools,  such  as  statistical 
analyses,  mathematical  modeling,  surveys,  and  interviews  in  her  analyses,  [riposo@rand.org] 

Other  co-contributors  from  RAND:  Jeffrey  A.  Drezner,  Douglas  Shontz,  Geoffrey  McGovern,  Daniel 
Tremblay,  Clifford  Grammich,  Jerry  M.  Sollinger,  Jason  Kumar 

Abstract 

Acquisition  data  underpin  the  management  and  oversight  of  the  U.S.  defense  acquisition 
portfolio.  However,  balancing  security  and  transparency  has  been  an  ongoing  challenge. 
Some  acquisition  professionals  are  not  getting  the  data  they  need  to  perform  their  assigned 
duties  or  are  not  getting  the  data  and  information  in  an  efficient  manner.  To  help  guide  the 
Office  of  the  Secretary  of  Defense  (OSD)  in  addressing  these  problems,  the  RAND 
Corporation  identified  access  problems  at  the  OSD  level — including  those  organizations  that 
require  access  to  data  and  information  to  support  the  OSD,  such  as  analytic  support  federally 
funded  research  and  development  centers  and  direct  support  contractors — and  evaluated  the 
role  of  policy  in  determining  access.  The  study  also  involved  a  limited  review  of  how  data  are 
shared  between  the  OSD  and  military  departments.  Issues  with  access  to  acquisition  data 
and  information  in  the  Department  of  Defense  (DoD)  finds  that  the  process  for  gaining  access 
to  data  is  inefficient  and  may  not  provide  access  to  the  best  data  to  support  analysis,  and  that 
OSD  analytic  groups  and  support  contractors  face  particular  challenges  in  gaining  access  to 
data.  Given  the  inherent  complexity  in  securing  data  and  sharing  data,  any  solutions  to 
problems  associated  with  data  sharing  must  be  well  thought  out  to  avoid  the  multitude  of 
unintended  consequences  that  could  arise. 

Introduction 

Acquisition  data  are  vast  and  include  such  information  as  the  cost  of  weapon 
systems  (both  procurement  and  operations),  technical  performance,  contracts  and 
contractor  performance,  and  program  decision  memoranda.  These  data  are  critical  to  the 
management  and  oversight  of  the  $1 .5  trillion  portfolio  of  major  weapon  programs  by  the 
Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 
(OUSD[AT&L];  GAO,  2014,  p.  3).  Data  collection  and  analysis  enable  the  Department  of 
Defense  (DoD)  to  track  acquisition  program  and  system  performance  and  ensure  that 
progress  is  being  made  toward  such  institutional  goals  as  achieving  efficiency  in  defense 
acquisition  and  delivering  weapon  systems  to  the  field  on  time  and  on  budget. 

Many  organizations  or  groups  need  access  to  this  information  for  a  variety  of 
purposes  (e.g.,  management,  oversight,  analysis,  and  administrative).  These  organizations 
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include  various  offices  of  the  DoD,  federally  funded  research  and  development  centers 
(FFRDCs),  university-affiliated  research  centers  (UARCs),  and  a  range  of  support 
contractors.  For  example,  an  FFRDC  may  need  cost  and  schedule  information  to  determine 
whether  a  weapon  system  was  delivered  on  time  and  within  budget.  Or  a  support  contractor 
may  be  responsible  for  managing  a  centralized  information  system  for  the  DoD  that  contains 
information  about  specific  procurement  programs.  Note  that  that  situation  does  not  include 
classified  data,  which  is  not  a  topic  of  this  report.1 

However,  these  organizations  may  have  difficulty  getting  access  to  these  data.  Some 
examples  of  the  types  of  issues  identified  by  individuals  within  DoD  offices  include  the 
following: 

•  “It  took  me  three  months,  multiple  e-mails,  and  phone  calls  to  get  a  one-hour 
meeting  with  five  SES  [DoD  senior  executive  service-level  employees]  to 
view  data  that  might  be  proprietary.” 

•  “Each  access  account  I  create  is  like  five  touch  points  between  an  email, 
phone  call,  their  POC,  certificate  handling,  vetting.  It’s  a  lot  of  work.” 

•  “If  there  are  dozens  of  support  contractors  and  dozens  of  prime  contractors 
and  I  have  to  get  an  NDA  [nondisclosure  agreement]  for  each  support 
contractor  and  prime  contractor  combination,  it’s  a  lot  of  work.” 

•  Examples  of  the  types  of  issues  identified  by  FFRDC,  UARC,  and  direct 
support  contractors  include 

•  “The  sponsor  has  to  have  access,  then  request  a  download  of  several 
documents  I  need,  then  transfer  the  data  to  me.” 

•  “I  couldn’t  get  access  because  I  didn’t  have  a  .mil  e-mail  address.” 

In  some  cases,  the  information  may  be  the  intellectual  property  of  a  commercial  firm. 
Sometimes  such  information  is  designated  proprietary.  This  information  requires  the 
permission  of  the  firm  that  owns  the  information  to  use  it.  The  process  of  getting  permission 
to  use  the  information  can  be  time-consuming,  may  never  yield  permission,  or  is  simply  too 
onerous.  An  example  of  the  third  possibility  is  a  database  that  has  proprietary  information 
from  many  firms,  requiring  support  contractors  to  sign  NDAs  with  each  firm,  which  could 
number  many  dozens  and  take  a  very  long  time. 

The  Office  of  the  Secretary  of  Defense  (OSD)  asked  the  RAND  National  Defense 
Research  Institute  to  identify  the  problems  and  challenges  associated  with  sharing 
unclassified  information  and  to  investigate  the  role  of  policies  and  practices  with  such 
sharing  in  the  first  phase  of  two  analyses  on  acquisition  data  (Riposo  et  al.,  2015).  In  the 
second  phase,  RAND  was  asked  to  evaluate  how  marking  and  labeling  Controlled 
Unclassified  Information  (CUI)  procedures,  practices,  and  security  policy  affect  access  to 
acquisition  oversight  data  (McKernan  et  al.,  2016).  We  will  present  the  approaches,  findings, 
and  options  for  improvement  for  both  analyses. 


1  Classified  information  is  any  information  designated  by  the  U.S.  government  for  restricted 
dissemination  or  distribution.  Information  so  designated  falls  into  various  categories  depending  on  the 
degree  of  harm  its  unauthorized  release  may  cause.  This  report  does  not  deal  with  classified 
information. 
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Phase  1  Approach 

We  pursued  a  three-pronged  approach  for  the  first  phase  of  this  research  with  the 
objective  of  defining  and  evaluating  any  data-sharing  problems.  The  first  part  of  the 
approach  was  a  policy  review.  We  began  by  reviewing  DoD  directives,  instructions, 
manuals,  and  guides,  along  with  executive  orders,  legislation,  and  regulations  concerning 
information  management.  The  objective  of  the  review  was  to  develop  a  framework  for 
understanding  what  governs  information  sharing  in  DoD  acquisition.  As  part  of  this  search, 
we  also  looked  at  a  limited  number  of  key  federal  policies  that  might  affect  data  sharing 
within  the  DoD. 

We  then  met  with  individuals  within  OSD  to  discuss  information  sharing,  which  is  the 
second  part  of  our  approach.  We  used  these  discussions  to  help  identify  information-sharing 
practices  and  issues  associated  with  data  access  and  releasability.  The  discussions  also 
helped  us  identify  relevant  policies  and  practices.  We  selected  a  sample  of  offices  within 
OUSD(AT&L)  to  reflect  a  variety  of  roles  in  the  acquisition  process.  We  spoke  with  data 
owners,  maintainers,  users,  and  individuals  involved  with  the  governance  of  information.  We 
categorized  the  offices  represented  in  the  sample  by  their  missions  and  roles.  This  step  led 
to  three  main  categories  of  OSD  offices: 

•  functional  and  subject-matter  experts 

•  Overarching  Integrated  Project  Team/Defense  Acquisition  Board  (OIPT/DAB) 
review  offices 

•  analysis  offices 

Within  the  OSD,  the  functional  and  subject-matter  experts  mainly  work  within  a 
specialty  (e.g.,  testing,  cost,  systems  engineering,  contracts,  earned  value).  Those  in  the 
OIPT  offices  are  primarily  responsible  for  direct  interaction  with  acquisition  programs  to 
review  portfolio  status  and  program  readiness  as  programs  move  through  the  acquisition 
process.  The  analysis  offices  conduct  a  variety  of  crosscutting  analyses  in  defense 
acquisition.  The  offices  that  fall  into  these  categories  appear  in  Table  1 .  We  also  interviewed 
service-level  acquisition  personnel  to  determine  the  role  that  the  services  play  in  DoD  data 
sharing. 

Our  goal  for  the  interviews  was  to  collect  the  following  information  regarding 
interviewees’  data  sharing  and  practices: 

•  role  in  the  acquisition  process 

•  data  needed  to  perform  one’s  job 

•  how  data  are  handled,  obtained,  and  provided  to  others 

•  data  access  or  release  problems 

•  data-sharing  recommendations 
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Table  1.  Offices  With  Roles  in  the  Acquisition  Process 


Office  Category 


Offices 


Functional  and  Subjecl-Matter  Experts *  *  QUSD(AT&L)  Performance  Assessments  and  Root  Cause  Analyses 

(PARCA)  Earned  Value  Managemenl  (EVM) 

■  OSD  Cost  Assessment  and  Program  Evaluation  (CAPE) 

■  OUSDCATSlL)  Human  Capital  Initiative  {HQ] 

■  OUSD(AT&L)  Defense  Procurement  and  Acquisition  Policy  {DPAP) 
-  OUSD(AT&L)  Developmental  Test  and  Evalualion  (DT> 

■  OUSD(AT&L)  Systems  Engineering  (SE) 


OIPT/DAB  Review  Offices  *  OUSD(ATSL)  Deputy  .Assistant  Secretary  of  Defense  (DASD}  Tactical 

Warfare  Systems  (TWS) 

■  OUSD(AT&L)  DASD  Space,  Strategic  and  Intelligence  Systems  (SSI) 
*  OUSD(ATSiL)  DASD  Command,  Control,  Communication,  Cyber  and 
Business  Systems  (C3CB) 


Analysis  Offices  *  OUSD(AT&L)  Acquisition  Resources  and  Analysis  (ARA) 

■  OUSD(ATSiL)  Defense  Acquisition  University  (DAU) 

-  DPAP 

■  FFRDCs 

-  OUSDCATStL)  PARCA  (outside  EVM) 


The  final  part  of  our  three-pronged  approach  for  phase  1  involved  conducting  two 
case  studies  to  illuminate  key  issues  and  challenges  associated  with  data  access.  Both 
reflect  (or  embody)  the  perception  of  several  key  data  access  issues.  The  first  case  study 
examines  the  use  of  proprietary  information  (PROPIN)  in  acquisition,  with  a  particular  focus 
on  earned  value  data.  The  second  looks  at  the  various  central  data  repositories  that  OSD 
maintains  and  uses.  More  specifically,  the  focus  was  on  the  background,  benefits,  and 
problems  associated  with  these  repositories.  During  our  introductory  interviews,  we  heard 
about  problems  with  using,  managing,  and  accessing  PROPIN  due  to  the  need  to  involve 
direct  support  contractors  in  the  collection  and  analysis  of  these  data.  Such  relationships 
require  the  use  of  NDAs  to  help  prime  contractors  and  subcontractors  protect  their 
information.  Both  case  studies  are  informed  by  the  interview  results  and  policy  analysis. 

Phase  2  Approach 

During  the  second  phase  of  this  analysis  on  acquisition  data,  we  evaluated  how 
marking  and  labeling  CUI  procedures,  practices,  and  security  policy  affect  access  to 
acquisition  oversight  data.  Our  work  for  this  phase  of  research  on  managing  and  handling 
acquisition  data  within  the  DoD  included  policy  analysis,  structured  discussions  with 
government  personnel,  and  a  literature  review  to  further  understand  and  evaluate 
proprietary  information  sharing,  the  origins  of  commonly  used  acquisition  labels,  and  how 
security  policy  affects  the  management  of  two  acquisition  information  management  systems 
within  the  OUSD(AT&L).  We  executed  our  work  through  three  main  tasks. 

•  Identify  and  evaluate  options  to  improve  nongovernment  employee 
access  to  proprietary  information:  We  continued  to  explore  the  source  of 
the  problems  identified  in  our  earlier  research  with  sharing  proprietary  data 
among  the  government,  contractor-originators  who  are  providing  the 
acquisition  information,  and  other  nongovernment  entities  such  as  federally 
funded  research  and  development  centers  (FFRDCs),  Systems  Engineering 
and  Technical  Assistance  (SETA)  support,  and  information  technology  (IT) 
support  contractors  who  are  supporting  the  government.  We  developed  a 
range  of  options  for  improving  direct  access  for  nongovernment  employees  to 
proprietary  data  and  documented  the  options  that  the  OUSD(AT&L)  is 
pursuing  to  improve  sharing.  We  characterized  the  options  and  their 
advantages  and  disadvantages  and  assessed  implementation  strategies  for 
them. 
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•  Characterize  commonly  used  data  markings  that  support  acquisition 
decision-making  and  oversight  and  identify  the  origins  of  those 
markings:  We  focused  on  CUI  labels  that  are  commonly  used  by  DoD 
government  and  nongovernment  employees  in  the  acquisition  process.  We 
identified  their  basis  in  law  and  policy  and  determined  whether  the  policy 
prescriptions  they  provide  for  data  labeling  and  access  are  clear  and 
consistent  and  accord  with  OUSD(AT&L)  goals.  OUSD(AT&L)  decision¬ 
making  and  oversight  is  intimately  connected  to  acquisition  data  access, 
research,  and  analysis.  Whether  these  data  are  available  for  timely, 
actionable  decision-making  partially  depends  on  the  type  of  data,  the  data 
control  system,  and  the  ability  of  data  users  to  properly  identify  and  label 
data,  and  if  necessary,  challenge  improperly  marked  data. 

•  Describe  how  DoD  security  policies,  processes,  and  procedures  affect 
OUSD(AT&L)’s  ability  to  provide  efficient  and  secure  access  to 
acquisition  data:  This  task  involved  multiple  steps.  First,  we  collected 
policies  that  affect  information  security  and  defense  acquisition  data  for  two 
information  systems  within  the  OUSD(AT&L) — Acquisition  Information 
Repository  (AIR)  and  the  Defense  Acquisition  Management  Information 
Retrieval  (DAMIR)  information  systems.  Second,  we  described  the  security 
policy  environment  for  managing  these  information  systems  (e.g.,  who  owns 
these  policies  and  what  topics  they  discuss).  Third,  we  described  and 
summarized  the  information  security  policy  and  identified  how  particular 
policies  affect  the  OUSD(AT&L)’s  ability  to  provide  access  to  acquisition  data 
and  manage  acquisition  data. 

Phase  1  Findings  and  Recommendations 

•  The  process  for  gaining  access  to  data  is  inefficient  and  may  not 
provide  access  to  the  best  data  to  support  analysis.  Government 
personnel  and  those  supporting  the  government  sometimes  do  not  get  their 
first  choice  of  data,  and  even  that  data  may  take  a  long  time  to  receive.  They 
may  be  forced  to  use  alternative  sources,  which  often  have  data  of  lower 
quality,  which  might  be  dated  and  thus  less  accurate,  or  be  subject  to  a 
number  of  caveats.  While  the  consequences  of  these  limitations  are 
undocumented  and  difficult  to  assess  and  quantify,  the  results  of  these 
analyses  can  be  inferior,  incomplete,  or  misleading. 

•  Two  groups  of  people  face  particular  challenges  in  gaining  access  to 
data:  OSD  analytic  groups  and  support  contractors.  OSD  analytic  groups 
often  do  not  have  access  to  the  originators  of  the  data,  which  precludes  them 
from  going  to  the  primary  source.  They  also  tend  to  have  poor  visibility  of  all 
viable  data  sources,  which  encourages  inefficient  data-seeking  practices. 
Direct  support  contractors  have  problems  similar  to  OSD  analysts,  but  these 
problems  can  be  compounded  by  laws,  regulations,  and  policy  that  restrict 
access  to  certain  types  of  information  (especially  nontechnical  proprietary 
data  that  originate  and  are  labeled  outside  the  government),  which  introduces 
extreme  inefficiencies.  Support  contractors  require  special  permissions  to 
view  nontechnical  proprietary  data. 

•  Difficulty  in  gaining  access  occurs  for  several  reasons: 

o  Data  access  policy  is  highly  decentralized,  not  well  known,  and 
subject  to  a  wide  range  of  interpretation. 
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o  The  markings  for  unclassified  information  play  a  significant  role  in 
access.  The  owner  or  creator  of  a  document  determines  what 
protections  or  markings  are  required.  However,  marking  criteria  are 
not  always  clear  or  consistently  applied.  In  fact,  management  and 
handling  procedures  for  many  commonly  used  markings  are  not 
clearly  described  anywhere.  Once  marked,  getting  the  labels  changed 
can  be  difficult.  When  information  is  not  marked,  the  burden  of 
handling  decisions  is  placed  on  the  receiver  of  the  information. 

o  Institutional  and  cultural  barriers  inhibit  sharing.  The  stove-piped 
structure  of  the  DoD  limits  visibility  and  sharing  of  data  and 
information.  Institutional  structure  and  bureaucratic  incentives  to 
restrict  data  access  are  exacerbated  by  policy  and  guidance  to  protect 
information.  The  result  is  a  strong  conservative  bias  in  labeling  and  a 
reluctance  to  share.  A  lack  of  trust  and  established  relationships  can 
hinder  sharing. 

Options  for  Improving  Data  Sharing 

The  variety  of  identified  problems  may  be  addressed  in  many  ways.  Each  potential 
option  requires  further  analysis  and  investigation.  We  offer  initial  thoughts  to  deal  with  the 
issue  of  access  to  proprietary  data,  as  well  as  the  general  confusion  regarding  policy. 

Options  to  Address  Problem  of  Proprietary  Data  Access 

There  are  several  potential  options  to  resolve  the  problem  of  access  to  proprietary 

data. 


•  The  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
(USD[AT&L])  could  seek  additional  billets  and  insource  any  functions  that 
require  access  to  proprietary  data.  However,  this  would  require  Office  of 
Personnel  Management  and  congressional  support. 

•  USD(AT&L)  could  seek  relief  through  a  reallocation  of  billets  to  functions  that 
currently  require  access  to  proprietary  information.  This  would  require  cross- 
organizational  prioritization,  a  difficult  process. 

•  General  access  could  be  established  for  all  direct  support  contractors.  This 
would  require  legislative  or  contractual  changes.  Current  legislation,  Title  10 
U.S.  Code,  Section  129d,  allows  litigation  support  contractors  to  view 
proprietary  information.  Similar  legislation  might  be  pursued  for  all  support 
contractors. 

•  Alternatively,  additional  contractual  language  could  be  placed  on  all  DoD 
acquisition  contracts  granting  support  contractors  restricted  access  to  their 
data.  The  direct  support  contractors  who  receive  the  data  would  have  to 
demonstrate  company  firewalls,  training,  personal  agreements,  and  need  to 
know  akin  to  those  for  classified  information. 

•  The  government  could  seek  an  alternative  ruling  on  the  nondisclosure 
requirements,  whereby  blanket  nondisclosure  agreements  could  be  signed 
between  the  government  and  a  direct  support  organization,  or  a  company 
and  a  direct  support  organization  to  cover  multiple  tasks. 

Each  of  these  options  would  require  further  analysis  and  coordination  with  Office  of 
the  General  Counsel  and  Defense  Procurement  and  Acquisition  Policy  (and  Congress  in  the 
first  and  third  options). 


mtSTANTIA  PER  SCIENTIAL 
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Options  to  Address  Policy  Confusion 

There  are  also  several  options  to  address  the  confusion  regarding  policy. 

•  OUSD(AT&L)  could  create  and  maintain  a  central,  authoritative  online 
resource  that  references  all  relevant  guidance  on  information  management, 
handling,  access,  and  release  for  acquisition  data.  This  would  require 
identifying  the  relevant  policy  and  posting  new  policies  as  they  become 
available. 

•  However,  an  online  resource  may  not  address  the  issue  of  the  workforce 
having  a  general  lack  of  expertise  and  insight  regarding  the  existing  policy 
and  guidance.  To  cope  with  this  problem,  OUSD(AT&L)  could  also  consider 
providing  additional  training  for  its  staff  on  the  identification  and  protection  of 
data.  This  could  be  an  annual  online  training  for  all  OUSD(AT&L)  staff  and 
contractors. 

•  In  areas  where  conflicting  interpretations  of  guidance  are  particularly 
problematic,  such  as  with  For  Official  Use  Only  (FOUO)  and  proprietary 
information,  additional  guidance  about  how  to  determine  whether  information 
is  FOUO  or  proprietary  in  the  first  place  would  be  helpful.  The  guidance 
should  provide  specific  examples  of  information  that  is  considered  protected, 
guidelines  for  determining  whether  specific  information  qualifies,  and  details 
regarding  handling  procedures  for  this  information,  to  include  access 
privileges. 

•  Directives  and  incentives  could  be  established  so  that  markings  that  appear 
to  be  incorrect  are  challenged  and  not  taken  only  on  a  company  or 
individual’s  claim.  If  more-detailed  determination  guidance  is  available,  it 
could  be  used  to  assess  the  validity  of  a  marking.  A  process  should  be  in 
place  for  challenging  markings,  and  it  should  be  exercised. 

There  are  important  reasons  for  restricting  access  that  require  balancing  control  with 
granting  more  access.  In  information  assurance  and  security  policy,  there  is  an 
understanding  that  no  individual  should  have  unfettered  access  to  all  data.  Given  the 
inherent  complexity  in  securing  data  and  sharing  data,  any  solutions  to  problems  associated 
with  data  sharing  must  be  well  thought  out  to  avoid  the  multitude  of  unintended 
consequences  that  could  arise. 

Phase  2  Findings  and  Recommendations 
Proprietary  Information  (PROPIN) 

PROPIN  is  a  special  class  of  CUI  that  relates  to  information  and  data  developed  by  a 
private  entity  but  shared  with  the  government.  Substantial  confusion  exists  within  the  DoD 
about  what  information  is  truly  proprietary,  who  can  have  access  to  it,  and  how  to  grant 
access  when  needed.  Despite  the  fact  that  some  policies  attempt  to  define  PROPIN  and 
handling  restrictions,  no  single  source  describes  the  processes  and  procedures  for  dealing 
with  this  type  of  information.  Rather,  a  patchwork  of  law,  regulation,  and  policy  govern  it, 
some  of  which  is  clear,  but  some  of  which  is  less  so.  This  hinders  the  DoD’s  use  of 
contractors,  restricts  information  flow,  and  limits  analyses. 

DoD  personnel  are  confused  about  who  can  access  PROPIN.  Information  so 
characterized  generally  can  be  treated  like  all  other  CUI,  meaning  all  government  personnel 
can  be  granted  access  (Treanor,  1999).  This  access  is  enabled  by  virtue  of  the  fact  that  the 
government  has  obtained  the  information  under  a  lawful  requirement.  Further,  federal 
employees  who  improperly  use  PROPIN  can  be  fired  and/or  prosecuted.  In  addition, 
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employees  with  a  security  clearance  sign  a  blanket  nondisclosure  agreement  (NDA) 
between  the  employee  and  the  government.  However,  many  government  personnel  are  not 
familiar  with  this  longstanding  practice  and  are  reluctant  to  share  information  with  other 
government  personnel  because  of  concerns  about  violating  an  unknown  law  or  regulation.  In 
addition,  procedures  for  nongovernment  personnel  to  gain  access  vary  widely.  Federal  law 
(10  U.S.C.  2320)  specifically  addresses  support  contractor  access  to  technical  data 
provided,  but  that  law  does  not  address  nontechnical  proprietary  information  supplied  by 
contractor-originators.  Consequently,  DoD  personnel  often  grapple  with  access  issues 
among  government  and  nongovernment  personnel  because  of  the  lack  of  clear  guidance 
about  who  can  access  what  information — and  what  information  constitutes  PROPIN. 

Ultimately,  the  company  submitting  the  information  to  the  government  is  responsible 
for  asserting  that  certain  portions  are  proprietary,  but  the  government  recipient  is 
responsible  for  determining  whether  to  accept  that  assertion  and  maintaining  the 
“proprietary”  label.2  In  other  words,  if  the  responsible  government  official  determines  the 
information  is  not  proprietary,  the  government  person  is  under  no  obligation  to  go  back  to 
the  company  (originator)  to  disclose  the  information  within  the  government  to  a  support 
contractor.  If  the  government  person  wants  to  publicly  disclose  the  information  in  response 
to  a  FOIA  request,  then  the  government  person  would  have  to  notify  the  company 
(originator).  However,  true  PROPIN  can  only  be  disclosed  within  the  government  to  support 
contractors  (and  now  FFRDC  employees)  when  a  one-to-one  (i.e.,  between  each  individual 
at  the  support  contractor/FFRDC  and  each  company  or  program  originating  data)  NDA  has 
been  executed. 

The  government  distinguishes  between  contractors,  generally,  and  the  special 
contractual  relationship  established  with  federally  funded  research  and  development  centers 
(FFRDCs).3  In  the  past,  the  special  relationship  has  meant  that  FFRDC  personnel  could  be 
granted  access  to  information  directly  by  government  personnel,  or  by  signing  a  single, 
blanket  NDA  between  the  employee  and  the  government,  allowing  them  access  to 
proprietary  information  in  the  course  of  their  government-related  work.  But  federal  law  does 
not  specifically  define  what  an  FFRDC  is  or  how  to  grant  FFRDC  personnel  access  to 
PROPIN.  Nontechnical  PROPIN  is  not  specifically  defined  in  statute,  and  courts  have  stated 
that  what  is  truly  proprietary  is  determined  on  a  case-by-case  basis  under  FOIA  Exemption 
4.  Generally,  the  disclosure  of  the  information  must  present  the  potential  for  a  company’s 


2  This  statement  is  based  on  the  researchers’  understanding  of  current  practices. 

3  FFRDCs  have  a  unique  relationship  with  the  government  because  they  have  access  beyond  that 
which  is  common  to  the  normal  contractual  relationship.  They  are  free  from  organizational  conflicts  of 
interest.  Also,  it  is  not  the  government’s  intent  that  an  FFRDC  use  its  privileged  information  or  access 
to  installations  equipment  and  real  property  to  compete  with  the  private  sector.  Finally,  FFRDCs  are 
meant  to  be  independent  research  institutions  characterized  by  objectivity.  According  to  48  C.F.R. 
35.017  (a.k.a.  FAR  35.017),  “An  FFRDC,  in  order  to  discharge  its  responsibilities  to  the  sponsoring 
agency,  has  access,  beyond  that  which  is  common  to  the  normal  contractual  relationship,  to 
Government  and  supplier  data,  including  sensitive  and  proprietary  data,  and  to  employees  and 
installations  equipment  and  real  property.  The  FFRDC  is  required  to  conduct  its  business  in  a  manner 
befitting  its  special  relationship  with  the  Government,  to  operate  in  the  public  interest  with  objectivity 
and  independence,  to  be  free  from  organizational  conflicts  of  interest,  and  to  have  full  disclosure  of  its 
affairs  to  the  sponsoring  agency.  It  is  not  the  Government's  intent  that  an  FFRDC  use  its  privileged 
information  or  access  to  installations  equipment  and  real  property  to  compete  with  the  private  sector.” 
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competitive  position  to  be  injured  by  a  competing  company  (Department  of  Justice,  2009,  p. 
305). 

Recent  DoD  interpretations  of  policy  and  statute — specifically  the  Trade  Secrets  Act 
(18  U.S.C.  1905) — have  changed  how  FFRDCs  are  treated  with  respect  to  NDAs,  resulting 
in  an  inefficient  and  ineffective  process  of  securing  them.  Specifically,  FFRDCs  are  now 
required  to  obtain  an  NDA  between  each  contractor-originator  of  data  in  a  system  and  each 
FFRDC  employee  who  needs  access — referred  to  in  this  report  as  “one-to-one”  NDAs. 
Previously,  FFRDC  employees  could  sign  a  single,  blanket  NDA  with  the  DoD  to  enable 
access  to  all  needed  information. 

The  RAND  Corporation  operates  three  FFRDCs:  Project  AIR  FORCE,  the  Arroyo 
Research  Center,  and  the  National  Defense  Research  Institute.  Therefore,  we  have  an 
interest  in  FFRDC  access  to  data.  We  believe  that  our  results  are  valid  independent  of  that 
interest,  and  we  have  firsthand  experience  with  the  struggles  of  DoD  personnel  managing 
data  and  access. 

Commonly  Used  CUI  Data  Markings 

The  current  set  of  CUI  labels  and  guidance  states  that  only  information  which 
requires  protection  by  Federal  Regulation  or  government-wide  policy  can  be  considered 
CUI.  In  other  words,  a  marking  that  does  not  originate  from  a  protection  established  by  law 
or  government-wide  policy  should  not  be  employed.  We  identified  nine  data  labels 
commonly  used  to  indicate  that  the  information  contained  in  a  document  or  database 
requires  some  type  of  special  handling  or  restriction.  Those  nine  labels  are 

•  Business  Sensitive 

•  Competition  Sensitive 

•  For  Official  Use  Only 

•  Pre-Decisional 

•  Proprietary 

•  Source  Selection  Sensitive 

•  Technical  Distribution  Statements 

•  DoD  Only 

•  Government  Only 

Some  of  these  labels  are  governed  by  well-established  policies  that  reflect  current 
understanding  of  the  law  and  regulatory  environment  for  data  protection  and  data  sharing. 
Others  are  legacy  markings  and  practices  that  were  not  aligned  with  draft  CUI  policy  at  the 
time  this  report  was  written.  We  were  unable  to  find  any  single  document  collecting  and 
describing  all  these  labels;  the  lack  of  a  single  such  document  contributes  to  the  general 
confusion  surrounding  them.  It  is  difficult  for  government  personnel  to  know  how  data  can  be 
shared.  A  result  of  this  situation  is  the  likely  over-labeling  and  mislabeling  of  CUI  material. 
Although  we  found  that  many  of  the  most  commonly  used  CUI  labels  do  have  a  basis  in  law 
or  policy,  labels  may  not  be  understood  in  practice,  used  properly,  or  have  clear  handling 
procedures. 

Consequently,  data  may  not  be  used  to  inform,  improve,  and  strengthen  the  DoD’s 
acquisition  functions.  Bottlenecks,  risk  aversion,  and  fear  of  releasing  otherwise  protected 
data  can  restrict  legitimate  access  and  data  sharing,  both  within  the  government  and 
between  the  government  and  select  partners.  While  the  National  CUI  program  being 
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established  by  the  National  Archives  will  help  provide  much-needed  clarifications,  it  is 
unclear  when  this  program  will  be  finalized  within  the  DoD. 

Implications  of  DoD  Security  Policies  for  Two  OUSD(AT&L)  Acquisition  Data 
Information  Systems 

Information  security  policies  directly  affect  the  access  and  utility  of  acquisition 
databases.  The  current  information  security  environment  does  not  establish  a  consistent 
framework  for  managing  information  systems.  This  makes  it  difficult  for  government 
employees  to  know  how  to  comply  with  regulations;  find  funds  and  the  technical  capabilities 
to  implement  new  policies;  develop  ways  to  evaluate  costs  and  benefits  of  new  policies  and 
determine  exceptions;  and  know  how  to  identify,  mark,  and  protect  CUI.  The  impact  of  these 
challenges  is  a  potential  delay  in  accessing  acquisition  data  by  both  government  and 
nongovernment  employees,  which  in  turn  may  result  in  lower  quality  analyses  or  decisions 
based  on  incomplete  information. 

We  used  the  Acquisition  Information  Repository  (AIR)  and  Defense  Acquisition 
Management  Information  Retrieval  (DAMIR)  OUSD(AT&L)  acquisition  data  information 
systems  as  case  studies  to  examine  the  implications  of  implementing  security  policies.  AIR 
provides  one  central  location  for  all  Major  Defense  Acquisition  Program  (MDAP)  and  Major 
Automated  Information  System  (MAIS)  acquisition  documents  to  support  oversight  and 
decision-making.4  DAMIR  fulfills  several  key  functions,  including  reporting,  storage,  quality 
assurance,  analysis,  oversight,  and  tracking  cost,  schedule,  and  performance  of  major 
acquisition  programs.5  AIR  largely  represents  the  unstructured  data  problem,  while  DAMIR 
represents  the  challenges  associated  with  structured  data  that  both  pull  from  and  feed  into 
other  information  systems. 

A  multitude  of  security  policies  affect  management  and  operation  of  these  systems. 
We  identified  about  two  dozen  executive  orders,  laws,  directives,  instructions,  operating 
guides,  and  other  policies  that  affect  AIR  and  DAMIR,  some  of  which  cover  similar  material. 
The  AIR  information  managers  have  created  a  set  of  business  rules  based  on  their 
interpretation  of  those  policies.  For  instance,  according  to  DoD  (2012)  Manual  5200.01, 
volume  4,  “The  [government]  originator  of  a  document  is  responsible  for  determining  at 
origination  whether  the  information  may  qualify  for  CUI  status,  and  if  so,  for  applying  the 
appropriate  CUI  markings”  (p.  9).  The  information  managers  for  AIR  have  interpreted  this 
policy  guidance  from  USD(I)  to  mean  that  the  originators  of  the  information  being  uploaded 
to  AIR  (e.g.,  the  services  and  other  OSD  offices)  are  responsible  for  appropriately  marking 
the  information  in  AIR  even  though  the  AIR  managers  have  noticed  some  inconsistency  in 
the  marking  of  the  documents  across  documents  types.  The  AIR  managers  attribute  this 
inconsistency  to  the  variety  of  security  classification  guides  being  used  to  mark  documents 
by  the  originators.  Also,  there  is  no  process  for  ensuring  that  up-to-date  marking 
conventions  are  followed  for  each  document  uploaded  to  AIR.  Management  and  use  of  AIR 


4  AIR  is  a  document  repository  that  contains  specific  program  documents  (reports,  certifications)  used 
to  inform  acquisition  decision-making  and  oversight. 

5  DAMIR  has  both  unclassified  and  classified  versions.  It  supports  the  generation,  distribution,  and 
archiving  of  Selected  Acquisition  Reports  (SARs)  as  well  as  information  supporting  the  Defense 
Executive  Acquisition  System  (DAES)  process.  It  also  includes  higher-level  earned  value 
management  data.  Unlike  AIR,  DAMIR  is  structured  data  that  users  can  combine  and  analyze  in 
multiple  ways  serving  multiple  functions. 
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are  complicated  by  the  need  to  access  it  on  an  IT  system  approved  through  Defense 
Security  Service  inspection,  use  a  .mil  e-mail  address  associated  with  a  Common  Access 
Card  (CAC),  and  have  approval  through  a  government  sponsor,  who  provides  the  rationale 
for  granting  a  user  access  to  AIR  for  a  specific  purpose.  In  addition,  the  permissions  process 
is  separate  from  the  sensitivity  of  documents  stored  in  AIR. 

DAMIR  is  hosted  by  the  Joint  Service  Provider,  which  only  partially  resides  within  the 
OUSD  (AT&L).  External  hosting  separates  operational  and  security  management  and 
creates  the  possibility  of  a  disconnect  between  the  business  case  for  data  use  and  security 
policies.  In  other  words,  the  cost  of  the  security  may  be  high  while  the  perceived  benefits 
may  be  low.  Understanding  the  business  case  (or  use)  for  DAMIR  is  critical  to  maintaining 
security  without  unduly  limiting  the  utility  of  the  system  for  users.  Security  policies  also  inhibit 
system  improvement,  which  requires  code  changes  and  upgrades.  A  recent  determination 
that  real  data  cannot  be  used  for  testing  required  additional  programming  work  to  invent 
data  to  test  the  system.  The  lack  of  actual  data  for  testing  makes  determining  whether  a  new 
database  capability  will  ultimately  work  a  speculative  exercise. 

Several  years  ago  a  security  policy  requiring  accounts  that  have  not  been  used  in  a 
30-day  period  to  be  disabled  significantly  affected  DAMIR.  Many  DAMIR  users,  including 
congressional  staff  and  FFRDC  analysts,  log  in  infrequently  (i.e.,  when  new  SAR  or  DAES 
reports  come  out)  rather  than  routinely.  The  policy  resulted  in  the  suspension  of  accounts, 
which  meant  the  DAMIR  team  had  to  re-register  about  30%  of  4,000  active  user  accounts 
initially  after  the  policy  was  enforced.  The  DAMIR  team  continued  to  have  significant 
problems  for  several  months  in  re-activating  inactive  accounts. 

Implementing  new  policies  within  DAMIR  (which  has  more  than  1.5  million  lines  of 
code)  is  also  challenging.  DAMIR  was  stood-up  under  different  security-related  policies,  and 
adapting  its  structure,  programming,  and  business  rules  to  accommodate  new  policies 
entails  substantial  effort.  Furthermore,  there  is  no  up-to-date  security  architecture  document 
because  architecture  and  security  policy  governing  DAMIR  have  evolved  independently. 
Similarly,  new  interpretations  of  existing  policies  have  consequences.  For  example,  a  new 
interpretation6  of  what  potentially  constitutes  personally  identifiable  information  (Pll)  caused 
the  DAMIR  management  team  to  conduct  a  formal  assessment  of  how  individual  privacy  is 
being  addressed  in  DAMIR  due  to  the  potential  existence  of  Pll  in  DAMIR. 

CUI  Marking  and  the  Security  Policy  Environment 

Overall,  the  current  environment  in  which  acquisition  data  are  protected  and  shared 
can  be  characterized  by  many  organizations  promulgating  policy  on  overlapping  and 
interrelated  topics,  policies  that  are  relatively  new  and  change  frequently,  and  an  ill-defined 
CUI  policy.  Furthermore,  security  policies  tend  to  be  one-size-fits-all,  which  does  not  reflect 
the  unique  characteristics  of  each  system.  Those  who  originate  the  policies  do  not  fund  their 
implementation,  meaning  that  a  new  or  changed  policy  is  effectively  an  unfunded 
requirement  for  system  managers.  This  situation  creates  a  number  of  issues  for  information 
system  managers.  First,  it  is  difficult  to  know  exactly  what  is  required  to  comply  with  the 


6  The  interpretation  was  based  on  the  reissue  of  DoD  Directive  (DoDD)  5400.1 1  that  updated  the 
established  policies  and  assigned  responsibilities  of  the  DoD  Privacy  Program  pursuant  to  section 
552a  of  Title  5,  U.S.C.  (also  known  and  referred  to  in  this  directive  as  “The  Privacy  Act”  and  Office  of 
Management  and  Budget  [OMB]  Circular  No.  A-130). 
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numerous  applicable  policies.  Second,  managers  have  to  find  the  funds  to  comply  when 
policies  change.  Third,  considerable  confusion  surrounds  the  identification  and  marking  of 
CUI.  This  environment,  which  is  causing  a  lot  of  inefficiency  and  many  workarounds  to  solve 
problems,  creates  a  managerial  problem  for  the  OUSD(AT&L). 

The  overall  effect  of  these  problems  almost  certainly  has  a  cost,  though  this  cost  is 
difficult  to  quantify.  Government  and  nongovernment  users  of  both  DAMIR  and  AIR  may,  for 
example,  simply  seek  to  conduct  analyses  with  other,  less  insightful  data,  or  without  data  at 
all.  No  system,  however,  tracks  the  effects  or  costs  of  DAMIR  and  AIR  (or  any  other 
information  system)  compliance  with  security  policy.  The  cumulative  effects  of  security  policy 
requirements  may  exceed  what  is  currently  documented  in  the  management  of  these  two 
acquisition  information  systems.  In  other  words,  the  effect  of  compliance  actions  on  other 
information  systems  and  user  behavior  can  have  a  cascading  effect;  the  problem  is  likely 
much  larger  than  what  has  been  documented  here. 

What  the  DoD  Can  Do  to  Improve  the  Situation 
Proprietary  Data 

We  suggest7  that  the  Federal  Acquisition  Regulation  (FAR)  FFRDC  provisions  could 
be  used  as  a  basis  for  a  DoD  decision  that  FFRDCs  are  exempt  from  the  relatively  new  one- 
to-one  NDA  requirement  created  by  a  change  in  DoD  interpretation  of  the  Trade  Secrets 
Act,  or  could  be  covered  by  a  single,  blanket  NDA  with  the  DoD.8  Office  of  Federal 
Procurement  Policy  staff  suggested  in  a  meeting  with  the  authors  of  this  report  that  the  DoD 
Office  of  the  General  Counsel  (OGC)  was  taking  an  overly  restrictive  view  of  the  FAR 
FFRDC  provisions.  For  non-FFRDC  contractors,  we  also  recommend  that  the  DoD  consider 
the  following: 

•  Creating  a  DFARS  provision  that  would  cover  nontechnical  data,9  possibly 
with  a  blanket  NDA  requirement 

•  Proposing  a  new  legislative  provision  covering  all  nongovernment  personnel 
similar  to  10  U.S.C.  129d,  which  allows  litigation  support  contractors  access 
to  “commercial,  financial,  or  proprietary  information”  without  a  nondisclosure 
agreement 

•  Proposing  a  legislative  amendment  to  10  U.S.C.  2320,  which  allows  access 
to  technical  data  for  providing  advice  or  technical  assistance  to  the 
government,  that  would  include  financial  and  management  data 

Regulatory  and  legislative  changes  both  carry  drawbacks.  The  DoD  can  propose 
changes  to  the  DFARS  without  congressional  action  and  presidential  approval,  but  changing 


7  Our  recommendations  are  designed  to  increase  access  to  sensitive  data  for  analysis.  As  a  party  that 
has  long  analyzed  such  data,  organizations  such  as  RAND  (an  FFRDC)  would,  of  course,  benefit 
from  such  actions,  and  we  understand  readers  may  view  our  recommendations  accordingly. 
Regardless,  we  trust  our  research  can  advance  broader  discussion  of  how  the  DoD  can  improve 
oversight  of  its  acquisition  programs. 

8  A  blanket  NDA  would  be  an  NDA  between  an  organization  and  another  organization,  versus  the 
current  requirement  of  a  one-to-one  NDA  between  an  individual  and  a  contractor-originator  of  data. 

9  As  noted  above,  10  U.S.C.  2320  specifically  addresses  technical  data,  so  we  are  only  discussing 
nontechnical  data. 
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the  DFARS  might  not  adequately  include  previous  PROPIN  designations  because  a  new 
clause  would  only  affect  contractors  who  presently  have  active  DoD  contracts.  Changing  the 
law  is  even  more  problematic  because  it  requires  congressional  action  and  presidential 
approval,  takes  approximately  two  or  more  years,  and  may  not  even  result  in  a  change  or 
could  result  in  unwanted  changes. 

CUI  Markings  and  Labels 

A  more  robust,  central  program  for  CUI  data  labeling,  access,  and  management 
(including  monitoring  and  challenging  document  originators)  may  help  facilitate  a  smoother 
sharing  and  protection  of  CUI  within  the  DoD.  The  DoD  should  also  train  its  workforce  on  the 
new  CUI  labeling  procedures  when  they  are  released  and  implemented  by  the  DoD.  Given 
that  no  central  reference,  institutional  structure,  or  authority  exists  for  defining  and 
establishing  proper  handling  procedures  for  CUI,  we  recommend  that  a  function  and 
reference  be  established  within  the  OUSD(AT&L)  for  both  technical  and  nontechnical 
acquisition  data. 

Security  Policy 

The  problem  that  needs  to  be  solved  with  respect  to  security  policy  is  the  clear 
mismatch  of  responsibility,  authority,  and  accountability  among  the  organizations  that  issue 
security  policy  and  manage  or  host  the  information  systems.  We  offer  several 
recommendations  oriented  at  addressing  this  problem. 

First,  we  suggest  using  existing  information  requirements  to  document  how  security 
policies  are  affecting  the  management  of  information  systems.  While  there  are  many 
anecdotes  about  difficulties  in  implementing  security  policy  for  AIR  and  DAMIR,  these  are 
not  documented  in  a  central  location  or  updated  over  time.  By  documenting  difficulties, 
including  resources  used  to  implement  various  policies,  the  OUSD(AT&L)  would  better 
understand  how  security  policies  are  affecting  their  systems  and  whether  a  better  balance 
between  security  and  business  cases10  is  being  achieved. 

Second,  we  suggest  that  a  function  be  established  within  the  OUSD(AT&L)  to  review 
information  security  policies,  de-conflict  them,  reduce  duplication,  ensure  consistency,  and 
identify  gaps  for  all  acquisition  data  collected  and  used  within  the  OUSD(AT&L).  This 
function  would  be  responsible  for  communicating  with  the  OUSD(AT&L)  information-system 
managers  in  order  to  have  a  greater  understanding  of  the  inefficiencies  in  implementing 
security  policy.  This  function  (or  working  group)  should  include  all  relevant  stakeholders  so 
as  represent  both  security  and  mission  perspectives. 

Third,  a  single  individual  should  be  designated  with  responsibility  for  implementing 
security  strategy  for  a  given  information  system.  This  individual,  the  AO,  could  work  with  the 
policy  originator  to  ensure  appropriate  interpretation  and  application  of  policy.  For  the 
OUSD(AT&L)  information  systems,  we  believe  that  the  AO  should  be  selected  based  on 
knowledge  of  the  mission  area  (i.e.,  a  subject  matter  expert).  The  goal  is  to  have  someone 


10  Enterprise  Information  within  OUSD(AT&L)/ARA  is  responsible  for  “providing  leadership  timely 
access  to  accurate,  authoritative  and  reliable  data  supporting  acquisition  oversight,  analysis,  and 
decision-making.”  El  needs  to  fulfill  its  mission  with  limited  resources,  so  it  must  balance  the  business 
case  for  adding  new  capability  to  its  information  systems  (DAMIR  and  AIR)  with  what  is  being 
mandated  for  it  to  implement  for  adequate  security  of  its  information  systems. 
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who  is  familiar  with  the  business  case  for  a  system  to  be  more  involved  in  the  daily 
operations  of  that  system  and  to  track  security  policy  changes  and  implementation. 

Fourth,  the  requirement  that  each  information  system  have  and  maintain  a  security 
strategy  should  be  used  as  an  opportunity  to  ensure  an  appropriate  balance  between 
security  risk,  business  case,  and  the  use  case11  for  each  information  system.  The  security 
strategy  should  be  updated  as  policies,  threats,  or  system  use  change,  providing  a 
consistent  framework  over  time  to  evaluate  the  balance  between  risk  and  utility. 

Finally,  implementation  of  security  policy  should  be  appropriately  resourced.  The 
issuing  organization  should  assess  required  resources  as  part  of  policy  design,  and  provide 
at  least  some  funding  to  address  needed  technical  changes  to  the  information  systems. 
Similarly,  the  organizations  managing  information  systems  should  identify  resources  to 
address  implementation  of  security  policy  as  part  of  the  security  strategy  it  maintains. 
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Abstract 

Acquisition  data  lay  the  foundational  role  for  decision-making,  management,  and  oversight  of 
the  weapon-systems  acquisition  portfolio  for  the  Department  of  Defense.  How  to  effectively 
and  efficiently  spend  these  dollars  has  been  a  top  priority  for  the  Better  Buying  Power 
initiatives  led  by  the  Office  of  the  Secretary  of  Defense  (OSD)  and  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics.  The  OSD  asked  RAND  to  help  identify 
how  available  data  can  help  assist  defense-acquisition  decision-making.  In  particular,  we 
documented  factual  information  on  21  information  systems  that  contain  acquisition  data.  This 
builds  on  our  earlier  work  (Riposo  et  al.,  2015,  Issues  With  Access  to  Acquisition  Data  and 
Information  in  the  Department  of  Defense:  Policy  and  Practice ,  RAND  RR-880;  and 
McKernan  et  al.,  2016,  Issues  With  Access  to  Acquisition  Data  and  Information  in  the 
Department  of  Defense:  A  Closer  Look  at  the  Origins  and  Implementation  of  Controlled 
Unclassified  Information  Labels  and  Security  Policy,  RAND  RR-1476)  by  exploring  in  more 
detail  the  data  that  support  decision-making. 

Introduction 

Acquisition  data1  lay  the  foundation  for  decision-making,  management,  and  oversight 
by  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics  (USD[AT&L]) 


1  Acquisition  data  are  vast  and  include  such  information  as  the  cost  of  weapon  systems  (both 
procurement  and  operations),  technical  performance,  contracts  and  contractor  performance,  and 
program  decision  memoranda.  These  data  are  critical  to  the  management  and  oversight  of  the  $1.5 
trillion  portfolio  of  major  weapon  programs  by  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics  (OUSD[AT&L]). 
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of  the  weapon-system  acquisition  portfolio  for  the  Department  of  Defense  (DoD).  Acquisition 
data  help  to  inform,  monitor,  and  achieve  several  DoD  objectives,  including 

•  promoting  transparency  in  spending 

•  understanding  and  achieving  cost  control 

•  visualizing  the  distribution  of  defense  spending 

•  achieving  small-business  goals 

•  identifying  and  preventing  fraud,  waste,  and  abuse 

•  conducting  analyses  for  improved  decision-making 

•  compiling  and  tracking  items  in  various  processes 

•  archiving  decisions 

It  is  critical  for  personnel  managing  acquisition  execution  and  oversight  to  know  what 
data  resides  within  DoD  as  well  as  what  questions  can,  or  cannot,  be  answered  with  that 
data  (Table  1). 

Table  1.  Acquisition  Data  Can  Answer  Some  Defense  Questions,  and  Not  Others 


(RAND) 

Acquisition  Data  Can  Answer 

Acquisition  Data  Cannot  Answer 

How  much  do  we  spend  acquiring  weapons 
and  services  in  DoD? 

Where  can  we  access  certain  technologies? 

How  long  does  it  take  to  field  a  weapon 
system? 

What  key  suppliers  may  need  help  given 
spending  decreases? 

What  are  the  key  costs  of  what  we  buy  for 
DoD? 

How  well  does  spending  align  with 
requirements? 

Who  do  we  buy  from? 

What  value  is  gained  from  spending? 

Where  is  our  spending  geographically 
located? 

How  is  the  health  of  the  defense  industrial 
base? 

Why  are  costs  and  schedule  increasing? 

What  is  the  quantity  adjusted!  cost  growth  for  a 
specific  acquisition  program? 

How  competitive  is  our  supply  base? 

How  do  the  acquisition  schedules  of  a  large 
number  of  acquisition  programs  compare  over 
time? 

How  well  does  DoD  meet  small  business 
goals? 

What  is  the  workload  of  various  parts  of  the 
acquisition  workforce  (e.g.,  contracting)? 

What  can  we  learn  from  past  acquisition 
failures/successes? 

Which  government  and  non-government 
personnel  are  working  specific  acquisition 
programs? 

How  to  effectively  and  efficiently  spend  taxpayer  dollars  allocated  to  the  Department 
of  Defense  has  been  a  top  priority  of  the  Better  Buying  Power  (BBP)  initiatives  led  the  Office 
of  the  Secretary  of  Defense  (OSD)  and  the  USD(AT&L).  In  BBP  2.0,  the  USD(AT&L) 
specifically  acknowledged  the  need  to  streamline  decision-making  by  “promptly  acquiring 
relevant  data  and  directing  differences  of  opinion  to  appropriate  decision-makers.  Our 
managers  cannot  be  effective  if  process  consumes  all  of  their  most  precious  resource — 
time”  (Kendall,  2013,  p.  2). 
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Currently,  much  weapon-system  acquisition  data  is  collected  based  on  policy 
directive,  congressional  reporting,  and  the  need  to  meet  USD(AT&L)’s  statutory  authorities. 
These  information  requirements  largely  reside  in  the  Department  of  Defense  Instruction 
(DoDI)  5000.02  (2015).  This  data-management  strategy  fails  to  address  the  complete 
managerial  prerogatives  of  the  USD(AT&L)  and  the  Better  Buying  Power  initiatives. 
Additionally,  siloed  reporting  of  acquisition  data  may  not  fully  support  the  USD(AT&L) 
decision-making  processes.  Data  requirements  have  generally  been  developed  from  a 
particular  functional  perspective  resulting  in  a  data  “ecosystem”  characterized  by  individual 
collections  of  data  that  are  functionally  stovepiped  and  disjointed,  each  with  different  rules 
for  collection,  retention,  and  access. 

Approach 

In  earlier  work  (Riposo  et  al.,  2015;  McKernan  et  al.,  2016),  we  identified  the  issues 
associated  with  managing  and  sharing  Controlled  Unclassified  Information  (CUI)  within  the 
DoD.  In  this  analysis,  we  examine  issues  with  managing  and  accessing  the  sources  of  that 
data.  Specifically,  the  OSD  asked  us  to  consider 

•  What  data  are  available  to  help  assist  in  defense  acquisition  decision¬ 
making? 

•  Where  do  acquisition  data  reside? 

•  Who  can  access  the  information? 

•  Can  we  get  access  to  these  data  for  acquisition-related  purposes? 

To  answer  these  questions,  we  held  targeted  discussions  with  acquisition  information 
system  managers,  supplemented  these  discussions  with  reviews  of  official  policy 
documentation  and  other  open  sources  on  the  information  systems  and  their  contents, 
reviewed  literature  on  master  data  management  to  understand  practices  in  commercial  data 
management,  and  augmented  our  findings  with  RAND  knowledge  of  using  these  data 
systems.  Through  these  methods,  we  accomplished  four  tasks. 

What  are  the  major  weapon  system  acquisition  data  domains?  We 

accomplished  this  task  by  reviewing  various  federal-wide,  OSD-wide,  and  Service-level 
information  systems  and  their  data  elements  in  order  to  identify  where  the  data  that  supports 
current  information  requirements  in  DoDI  5000.02  reside.  We  focused  first  on  a  broad  look 
at  the  enterprise  acquisition  landscape  as  a  whole,  then  particularly  on  sources  of 
acquisition  information  that  support  the  USD(AT&L)  through  the  Defense  Acquisition 
Executive  Summary  (DAES)  and  Defense  Acquisition  Board  (DAB)  secretariat,  Director, 
Acquisition  Resources  and  Analysis  (D,  ARA).  Our  sponsor,  deputy  director  of  Enterprise 
Information,  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics,  Acquisition  Resources  and  Analysis  Directorate,  provided  the  list  of  information 
systems  to  examine  for  this  analysis. 

What  are  the  functional  communities  or  major  users  that  weapon  system  acquisition 
data  domains  support  within  the  DoD?  We  identified,  through  discussions  with  the 
information  managers  of  the  21  information  systems,  major  users  of  DoD  acquisition  data 
within  the  OSD. 

What  are  the  providers  of  weapon  system  acquisition  data  for  USD(AT&L) 
decision-making?  We  also  identified,  through  discussions  with  information  managers,  who 
is  providing  acquisition  data  to  OSD  information  systems  in  order  to  inform  USD(AT&L) 
decision-making  on  defense  acquisition. 
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What  are  some  recommendations  for  improving  the  acquisition  data 
environment?  In  this  task,  we  provide  recommendations  that  would  improve  the  quality  of 
acquisition  data,  ease  of  access,  efficiency  of  collection  and  use,  and  the  ability  to  link  data 
through  common  data  elements. 

Background  on  Acquisition  Data  in  the  Department  of  Defense 

Acquisition  data  and  information  take  on  a  wide  variety  of  forms  within  the 
Department  of  Defense  and  include  such  information  as  the  cost  of  weapon  systems  (both 
procurement  and  operations),  technical  performance,  contracts  and  contractor  performance, 
and  program  decision  memoranda.  These  data  can  be  characterized  as  both  “structured” 
and  “unstructured.”2  They  are  critical  to  the  management  and  oversight  of  the  $1 .5  trillion 
portfolio  of  major  weapon  programs  by  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics. 

This  data  may  be  for  statutory,  regulation,  policy,  or  other  reasons.  DoDI  5000.02 
(2015,  Enclosure  1,  pp.  47-58)  provides  a  detailed  list  of  “statutory  and  regulatory 
requirements  at  each  of  the  milestones  and  other  decision  points  during  the  acquisition 
process.”  This  does  not  encompass  all  of  the  requirements,  but  is  a  centralized  source  for 
many  of  them.  Some  of  the  information  requirements  are  to  measure  cost,  schedule,  and 
performance  of  weapon  systems,  while  others  examine  testing,  cybersecurity,  requirements, 
budgeting,  alternatives,  and  technology  readiness. 

The  information  resides  throughout  the  DoD  at  all  levels,  from  program  offices  in  the 
Services  to  various  offices  within  OUSD(AT&L).  It  can  be  found  in  decentralized  locations 
(e.g.,  individual  computers)  and  centralized  locations  (e.g.,  information  systems).  The  DoD 
also  uses  data  that  reside  in  various  federal  information  systems.  There  is  a  plethora  of 
acquisition-related  data  sources  that  are  now  available.  The  data  elements  within  these 
information  systems  vary.  Some  data  elements3  are  unique  while  others  may  overlap, 
depending  on  different  definitions.  The  timeframes  for  the  various  data  elements  are  non¬ 
stationary,  meaning,  for  example,  that  one  information  system  has  data  from  1960  to 
current,  while  another  may  only  have  data  from  2010  to  current.  Acquisition  data  are  stored 
in  information  systems  with  differing  platforms  and  hardware:  architectures,  software,  and 
interfaces;  vendors;  and  databases.  There  is  varying  accessibility  and  security  requirements 
(depending  on  the  data  being  stored)  in  the  information  systems. 

Enterprise  Information  within  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics,  Acquisition  Resources  and  Analysis  Directorate 
categorizes  the  data  into  various  business  areas  including  Research  and  Development 


2  According  to  the  PC  Magazine  Online  Encyclopedia,  structured  data  are  “Data  that  can  be 
immediately  identified  within  an  electronic  structure  such  as  a  relational  database.”  Unstructured  data 
are  “Data  that  are  not  in  fixed  locations.  The  term  generally  refers  to  free-form  text  such  as  in  word 
processing  documents,  PDF  files,  e-mail  messages,  blogs,  Web  pages  and  social  sites”  (“Structured 
Data,”  n.d.-b). 

3  According  to  the  PC  Magazine  Online  Encyclopedia,  a  data  element  is  “The  fundamental  data 
structure  in  a  data  processing  system.  Any  unit  of  data  defined  for  processing  is  a  data  element;  for 
example,  ACCOUNT  NUMBER,  NAME,  ADDRESS  and  CITY.  A  data  element  is  defined  by  size  (in 
characters)  and  type  (alphanumeric,  numeric  only,  true/false,  date,  etc.).  A  specific  set  of  values  or 
range  of  values  may  also  be  part  of  the  definition”  (“Data  Element,”  n.d.-a). 
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(R&D),  Requirements,  Budget,  Contracting,  Contract  Performance,  Financial  Execution, 
Program  Cost/Schedule/Performance,  Human  Capital,  and  Acquisition  Oversight/Portfolio 
Management. 

Many  factors  affect  how  acquisition  data  is  collected  and  stored.  There  are  multiple, 
changing  conditions  that  affect  the  management  of  acquisition  data.  Information  owners  and 
managers  may  need  to  consider  whether  a  current  architecture  can  support  additional 
statutory  requirements,  administrative  changes,  or  security  policy  changes.  Technological 
advancements  may  also  be  implemented  to  improve 

•  Collection  efficiency 

•  Quality  of  the  data 

•  Aggregation  of  the  data 

•  Ease  of  access/use  of  the  information  system  and  its  data 

•  Analysis  of  data 

•  Archiving  data  for  future  analysis/education 

These  same  factors  can  also  affect  the  development  of  various  acquisition  systems. 
Acquisition  information  systems  were  created,  evolved,  or  repurposed  based  upon  data 
needs  and  legitimate  reasons  (e.g.,  statutory  needs).  They  have  been  developed  with 
varying  architectures  and  interfaces.  They  also  require  analysts  with  cross  system-analytic 
skills.  They  are  also  difficult  for  users  to  navigate  effectively,  and  can  takes  years  of 
consistent  access  and  use  to  fully  understand  and  master.  Most  systems  are  built  for 
reporting,  not  analysis.  Compliance  and  tracking  has  been  a  priority.  Acquisition  information 
systems  and  the  data  they  contain  may  be  designed  to  answer  today’s  current  questions, 
but  inflexible  to  answer  tomorrow’s  questions. 

This  analysis  found  that  there  are  also  barriers  to  the  use  of  each  system  and  cross 
use  between  the  information  systems.  Access  procedures  are  complicated  and  generally 
consist  of  many  steps  that  may  not  ultimately  guarantee  access.  There  are  varying  access 
procedures  and  permissions  between  and  sometimes  within  systems.  The  federal  systems 
have  much  data  available  to  the  public,  but  DoD  systems  are  mostly  restricted.  New  users 
can  have  great  difficulty  establishing  and  maintaining  access  (how  to,  where,  who,  what?). 
Full  access  to  acquisition  information  systems  enables  analysts  to  maximize  use  of  data. 
The  owners  and  managers  of  the  data  have  found  that  balancing  security  and  access  needs 
is  difficult. 

Background  and  Findings  on  Deep  Dives  of  Acquisition  Information  Systems 

As  part  of  this  effort  to  understand  acquisition  data  opportunities,4  we  conducted 
“deep  dives”  on  a  set  of  information  systems.  In  this  section,  we  summarize  the  information 
we  gathered  through  our  deep  dives.  We  reviewed  21  federal-wide,  OSD-level,  and  Service- 
level  information  systems  and  their  data  elements  in  order  to  identify  where  are  some  of  the 
acquisition  data  or  information  that  supports  current  requirements  in  DoDI  5000.02.  We 
reviewed  five  federal-level  information  systems,  12  OSD-level  information  systems,  and 


4  By  “data  opportunities,”  we  mean  identifying  data  that  can  potentially  be  used  for  analysis  of  various 
defense  acquisition  questions. 
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three  Service-level  systems  (one  Army,  one  Air  Force,  and  one  Navy).  Of  the  21  systems,  at 
least  one  study-team  member  had  previous  knowledge  of  1 1 .  For  five  systems,  a  study- 
team  member  had  limited  prior  or  current  knowledge;  and  for  the  final  five  systems,  no  one 
from  the  RAND  study  team  had  knowledge  from  use.  We  worked  with  our  sponsor,  ARA/EI, 
on  whether  to  pursue  access  to  the  information  systems  for  this  effort,  ultimately  deciding 
not  to  do  so. 

We  did  not  rely  exclusively  on  access  to  the  information  systems  in  order  to  conduct 
the  deep  dives.  We  also  collected  official  documentation  as  available,  and  requested 
additional  materials  from  those  managing  the  information  systems.  We  had  some  level  of 
open-source  materials  for  all  but  two  systems.  Finally,  we  relied  heavily  on  discussions  with 
the  information  managers,  particularly  on  the  information  systems  for  which  we  had  little  or 
no  knowledge  and  open-source  materials  were  not  available.  We  were  able  to  conduct 
discussions  for  all  but  one  information  system.  The  results  of  this  study  depend  on  the 
variety  of  information  we  were  able  to  collect. 

We  verified  the  deep-dive  information  with  information  managers  in  early  2016  in 
order  to  ensure  that  the  deep  dives  contain  the  latest  available  information.  Nevertheless, 
we  found  that  the  information  in  these  systems  is  constantly  changing  as  policy,  technology, 
and  other  things  change.  Consequently,  it  is  best  to  consult  the  information  systems  directly 
for  the  most  up-to-date  information. 

As  stated  previously,  we  gathered  additional  information  for  these  deep  dives 
through  discussions  with  information  managers.  The  information  that  we  gathered  from  the 
discussions  covered  the  following  main  topic  areas: 


•  Basic  details  on  the  acquisition  information  system 

•  Types  of  questions  answered  with  this  information  system 

•  Owner,  manager,  and  host  of  the  information  system  and  data  in  that 
information  system 

•  Statute  or  policies  that  led  to  the  creation  of  the  information  system  or  provide 
the  reason  the  data  in  the  system  is  collected 

•  Characterization  of  the  data  in  the  information  system 

•  Security  and  access  restrictions  governing  the  information  system 

•  Characterization  of  the  users 

•  Strengths  and  weaknesses  of  the  information  system  or  data  in  that 
information  system 


Basic  Details  on  the  Acquisition  Information  Systems 

For  each  of  the  21  information  systems,  we  gathered  basic  factual  information 
including  the  official  abbreviation,  date  that  the  system  entered  service,  the  access  point  for 
the  information  system,  whether  the  system  is  open  to  the  public  or  is  restricted,  the 
functional  business  area  the  system  supports,  and  the  purpose.  These  systems  cover  a 
wide  variety  of  functional  business  areas  including 


•  Research  and  development  (R&D) 

•  Requirements 

•  Budgeting 

•  Contracting 

•  Contract  Performance 
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•  Financial  Execution 

•  Program  Cost  /Schedule/Performance 

•  Human  Capital 

•  Acquisition  Oversight/Portfolio  Management 

Some  systems  cover  multiple  business  areas. 

Types  of  Questions  Answered  by  These  Information  Systems 

Decision-makers  and  analysts  working  in  defense  acquisition  need  to  understand  the 
type  of  questions  that  can  be  answered  with  the  structured  and  unstructured  data  in  these 
information  systems.  They  also  need  to  know  what  questions  cannot  be  answered.  We 
asked  information  managers  to  identify  some  of  the  questions  that  can  be  answered  from 
the  data  in  these  information  systems. 

Owner,  Manager,  and  Host  of  the  Information  System 

Additional  factual  information  that  we  collected  on  these  information  systems 
included  the  owner,  manager,  and  host  of  these  systems.  The  owner  is  the  office 
responsible  for  oversight  of  the  information  system.  It  is  sometimes  different  from  the 
manager  of  the  system  who  may  be  responsible  for  day-to-day  operations  including 
approving  access  and  troubleshooting  technical  issues,  but  the  owner  and  manager  are 
typically  within  the  same,  larger  organization.  The  host  of  the  information  system  often 
appears  to  be  an  office  outside  of  the  owner  or  manager  and  is  typically  a  contractor  for  the 
federal  systems. 

Statute/Policies  Requiring  Each  Information  System 

Most  of  these  systems  originated  in  statute  requirements,  with  the  Federal 
Acquisition  Regulation  also  being  a  common  reason  for  creating  a  data  system.  Some 
systems  originated  in  policies  or  memoranda  from  senior  DoD  leadership. 

Characterization  of  the  Data  in  the  Information  System 

There  is  no  consensus  on  whether  the  data  in  these  systems  is  authoritative.  Some 
systems  contain  data  that  are  authoritative,  but  others  pull  data  from  elsewhere.  There  is 
also  significant  variation  in  the  dates  of  the  data  in  these  information  systems.  A  version  of 
one  information  system  goes  back  as  far  as  1951  for  the  DoD.  For  several  other  systems, 
there  may  be  some  historical  data  back  to  the  1960s.  Likewise,  there  is  some  variation  in 
whether  a  formal  data  dictionary  exists  and,  if  one  does,  whether  it  is  available  to  users.  In 
some  cases,  information  managers  use  the  data  dictionary  for  planning,  but  do  not  provide  it 
to  users.  In  some  systems,  data  elements  have  been  added  over  time  or  their  definitions 
have  changed. 

Characterization  of  the  Users 

The  number  of  users  for  these  information  systems  varied  from  less  than  100  to 
nearly  400,000  users.  Information  managers  may  count  their  users  as  “registered,”  “active,” 
“average  users  per  month,”  or  “number  of  users  in  a  particular  time  period.”  Composition  of 
users  also  varies  widely.  Some  of  the  information  managers  provided  high-level  statistics 
(e.g.,  public,  government,  DoD),  while  others  provide  specific  organization  names  for  users. 

Conclusions  and  Options 

Acquisition  data  and  information  take  on  a  wide  variety  of  forms  within  the 
Department  of  Defense  and  include  such  information  as  the  cost  of  weapon  systems  (both 
procurement  and  operations),  technical  performance,  contracts  and  contractor  performance, 
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and  program  decision  memoranda.  This  data  is  collected  for  a  variety  of  reasons  including 
statutory  requirements,  regulation,  policy,  and  other  reasons. 

The  information  resides  throughout  all  levels  of  the  DoD  and  can  be  found  in 
informal,  decentralized  locations  as  well  as  formal,  centralized  locations  (e.g.,  information 
systems).  The  DoD  also  uses  other  federal  data  residing  elsewhere. 

Data  elements  within  this  plethora  of  sources  may  vary.  Some  data  elements  are 
unique,  while  others  may  overlap  depending  on  different  definitions.  The  timeframe  and 
source  of  these  data  vary  as  well. 

There  are  multiple,  changing  conditions  that  affect  the  management  of  acquisition 
data.  Information  owners  and  managers  may  need  to  consider  whether  a  current 
architecture  can  support  additional  statutory  requirements,  administrative  changes,  or 
security  policy  changes.  Technological  advancements  may  also  be  implemented  to  improve 
collection  efficiency,  quality,  aggregation,  and  ease  of  access  or  use. 

These  same  conditions  can  also  affect  the  development  of  various  acquisition 
systems.  Acquisition  information  systems  were  created,  evolved,  or  repurposed  based  upon 
data  needs  and  legitimate  reasons  (e.g.,  statutory  needs).  Yet  they  are  often  difficult  for 
users  to  navigate  effectively  and  can  require  years  of  consistent  access  and  use  to  fully 
understand  and  master.  Most  systems  are  built  for  reporting,  not  analysis,  and  compliance 
and  tracking  has  been  a  priority.  Acquisition  information  systems  and  the  data  they  contain 
might  answer  current  questions  but  may  be  inflexible  for  future  ones. 

This  analysis  found  that  there  are  also  barriers  to  use  of  each  information  system 
and  cross  use  between  the  information  systems.  Access  procedures  are  complicated  and 
generally  have  many  steps  that  need  to  be  met  in  order  to  be  permitted  access  to  the 
information  system  and  its  comments.  There  are  also  varying  access  procedures/ 
permissions  between  and  sometimes  within  systems.  The  federal  systems  have  an 
abundance  of  data  available  to  the  public,  but  DoD  systems  are  mostly  restricted.  New 
users  can  have  great  difficulty  establishing  and  maintaining  access.  Although  full  access  to 
acquisition  information  systems  enables  analysts  to  maximize  use  of  data,  it  is  not  practical 
given  the  need  to  balance  security  and  access. 

Deep  Dive  Conclusions 

We  compiled  information  on  21  federal  and  DoD  information  systems  that  contain 
structured  and  unstructured  acquisition  data  and  information.  The  level  of  detail  we  were 
able  to  pull  together  on  each  information  system  and  its  contents  varied  considerably  based 
on 


•  RAND  team  user  experience  with  individual  systems 

•  Availability  and  access  to  official  policy  documentation  and  other  materials  on 
the  information  systems 

•  Interviewee  interpretation  of  discussion  questions 

There  was  a  wide  variety  of  interpretation  of  each  of  the  questions  in  the  interview 
protocol  and  how  these  questions  pertain  to  the  individual  information  systems  that  an 
information  manager  is  overseeing.  The  output  of  these  discussions  showed  that  even 
common  terms  like  “owner,”  “user,”  or  “data  element”  and  “data  dictionary”  are  subject  to 
interpretation,  which  suggests  that  a  common  taxonomy  would  be  difficult  to  implement,  but 
may  be  necessary.  Basic  details  were  fairly  easy  to  identify  and  verify.  We  also  pulled 
together  a  large  variety  of  potential  questions  that  can  be  answered  by  the  data  in  each 
information  system,  but  the  list  is  not  comprehensive  nor  an  assessment  of  how  well  the 
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questions  could  be  answered.  Nevertheless,  both  are  critical  information  for  decision¬ 
makers. 

Some  factual  information  can  be  difficult  to  assess,  given  subtle  distinctions  such  as 
those  between  owner  and  manager  in  some  cases.  In  other  cases,  it  was  easy  to  verify 
information  on  owners,  managers,  and  hosts,  because  all  three  functions  are  performed  by 
the  same  office.  Yet  some  owners,  managers,  and  hosts  changed  over  time,  so  it  was  not 
always  clear  who  held  which  role. 

The  list  of  policies  that  led  to  the  origins  of  these  systems  was  not  always  apparent 
as  some  of  the  systems  are  older,  some  systems  have  “morphed”  from  one  objective  to 
others,  and  there  has  been  a  turnover  in  personnel  who  manage  the  systems.  Some 
information  systems  provided  a  list  within  the  information  system  documenting  the  policies 
that  led  to  the  system  creation/the  data  in  the  system.  In  other  cases,  we  were  given  the 
information  during  our  discussions  with  information  managers. 

When  we  asked  about  security  and  access  and  the  user  base  with  information 
managers,  the  feedback  we  got  was  very  difficult  to  compare  across  systems.  Security  and 
access  were  intertwined  in  discussions  even  though  there  are  supposed  to  be  clear  origins 
in  statute  and  policy  that  require  both  security  and  access  restrictions.  Similarly,  the 
information  we  received  on  users  varied  by  number,  type,  and  characteristic. 

For  each  data  system  we  reviewed  we  also  sought  to  identify  strengths  and 
challenges  for  the  information  manager  and  users.  We  summarized  the  major  cross-cutting 
strengths  and  challenges  themes  associated  with  the  systems  reviewed.  The  following  are 
some  of  the  major  strengths: 


•  The  collection  and  standardization  of  selected  acquisition  related  information 
into  one  place  where  it  can  be  input,  accessed,  and  analyzed  by  those 
needing  to  use  it. 

•  Data  that  is  input  electronically  with  controls  (e.g.,  through  validation  checks 
and  business  rules)  to  assure  that  key  data  elements  are  entered,  edited,  and 
cross  checked  against  historical  and  other  data,  which  improves  data  quality. 

•  Systems  that  have  been  established  or  improved  to  answer  acquisition 
questions.  These  systems  are  attempting  to  pull  together  variables  in  one 
place  for  analysis,  so  as  to  improve  DoD  decision-making,  and  also  to  save 
funding  that  is  typically  spent  by  analysts  trying  to  cobble  together 
information. 


Information  managers  also  face  several  challenges  in  managing  acquisition  data, 
including  the  following: 


•  Data  quality  vary  depending  on  what  is  input  or  provided,  and  often  with  no 
means  to  verify  accuracy. 

•  The  need  to  have  the  originators  input  new  data  when  the  data  have 
changed. 

•  Assuring  access  to  those  who  need-to-know  while  protecting  sensitive  data. 
Access  procedures  vary  greatly  by  system,  burdening  those  needing  to 
access  multiple  systems. 

•  Inconsistency  in  terms.  The  same  term  can  have  different  meanings  in 
different  acquisition  systems  which  makes  analyses  across  systems 
particularly  challenging. 

•  Inconsistency  in  data  formats. 
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•  High  variance  in  hardware  and  software. 

•  Need  for  more  data  elements,  leveraging  of  authoritative  systems,  real  time 
editing  and  verification,  and  updating  to  new  platforms. 

•  Desired,  backlogged  improvements  and  sometimes  critical  updates  that  lack 
resources  for  implementation. 

Options  for  Improving  the  Acquisition  Data  Environment 

Our  analysis  yields  several  recommendations  for  improving  the  DoD  acquisition-data 
environment. 

Formalize  a  Data  Governance  and  Data  Management  Function 

To  answer  the  DoD’s  acquisition  questions,  the  USD(AT&L)  should  consider 
formalizing  a  data  management  and  governance  function  (e.g.,  data  steward)  to  oversee 
data  opportunities.  Any  decision  on  a  data  steward  would  need  to  consider  who  could  be  the 
authority  to  institutionalize/implement  these  changes  given  the  diversity  of  data  ownership  in 
the  DoD. 

Our  discussions  with  information  managers  and  our  literature  review  on  Master  Data 
Management  found  that  data  governance  plays  a  key  role  in  the  success  of  acquisition  data 
management.  In  particular,  data  governance  can  monitor  and  enforce  the  use  of  acquisition 
tools.  Data  governance  also  determines  the  process  and  structure  for  authority  control, 
planning,  monitoring,  and  enforcement  over  data  assets  (American  Institute  of  CPAs,  2013, 
p.  4).  While  data  quality/validation  focuses  on  managing  individual  pieces  of  data,  data 
governance  focuses  on  data  definitions,  policies,  and  processes,  including  those  for  data 
quality/validation.  Data  governance  has  two  primary  data-management  objectives:  planning 
and  supervision/control. 

A  data  steward  function  would  need  to  further  identify  where  and  what  data 
opportunities  exist  by  maintaining  a  master  list  of  data/information  and  authoritative  sources. 
As  can  be  seen  from  this  study,  authoritative  sources  are  not  always  integrated  into 
information  systems,  and  it  is  not  apparent  that  developers  have  a  good  understanding  of  all 
of  the  authoritative  sources.  There  appears  to  be  a  movement  in  that  direction,  but  the  DoD 
should  continue  to  re-syndicate  data  from  authoritative  sources. 

The  data  steward  and  information  managers  should  proactively  solicit  ways  to 
improve  value  of  the  data  from  all  categories  of  users  (inputters,  overseers,  and  analysts)  in 
order  to  improve  data  quality,  capability,  access,  usability,  and  functionality.  This  function 
could  also  improve  understanding  of  related  systems  and  identify  potential  opportunities  for 
consolidation. 

Improve  Data  Quality  and  Its  Analytic  Value 

The  DoD  should  require  that  all  new  systems  have  user  and  data  entry  guides  and 
data  dictionaries  that  describe  data  elements  and  their  sources  (e.g.,  another  system  or 
enterprise/personnel  entering).  This  informs  data  opportunities  and  may  eliminate 
duplication.  Information  managers  should  try  to  minimize  manual  entry  whenever  possible  or 
provide  validation  checks.  An  explicit  list  of  authoritative  sources  for  data  elements  should 
be  available  and  new  systems  should  be  required  to  use  them,  while  older  systems  migrate 
towards  them. 

Information  managers  frequently  mentioned  that  data  verification  and  validation  is  a 
top  priority  and  that  they  have  both  manual  and  automated  checks  built  into  the  systems. 
Information  managers  should  continue  and  expand  this  best  practice. 
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Information  managers  mentioned  one  of  their  challenges  is  to  be  able  to  continue  to 
update  their  systems  to  add  capability  and  comply  with  the  latest  security  requirements.  The 
DoD  should  require  system  owners  to  develop  and  update  plans  and  budgets  for  continuous 
improvement  of  data  quality  and  analytic  value,  and  document  unfunded  requirements 
linked  to  these  improvements. 

Make  Structured  Data  the  Top  Priority 

Current  practice  is  to  collect  DoD  Acquisition  data  in  structured  and  unstructured 
formats.  Both  types  of  formats  have  an  important  role  in  the  execution,  oversight,  and 
analysis  of  acquisition  programs.  However,  structured  data,  which  is  easier  to  use  for 
analysis,  should  be  the  top  priority.  The  DoD  should  minimize  the  use  of  unstructured  data, 
which  takes  more  resources  and  different  capabilities  to  make  useful  for  analysis.  More 
specifically,  structured  data 

•  allows  for  topic  metatags 

•  can  use  strategic  algorithms  to  check  quality 

•  maximizes  drop-down  menus;  minimizes  free  text 

Similarly,  a  large  amount  of  acquisition  information  is  produced  in  unstructured 
formats.  Since  not  all  data  can  be  converted  to  a  structured  format,  the  DoD  needs  to 
identify  ways  to  make  unstructured  data  more  useful.  Structured  data  is  easy  to  use  once 
meaning  and  access  has  been  determined. 

By  moving  toward  structured  data,  the  standardization  of  formats  for  acquisition  data 
would  promote  sharing  between  systems.  The  standardization  needs  to  take  into  account 
context  and  meaning  when  appropriate. 

Develop  and  Train  Organic  Capability  Among  the  DoD  Workforce  to  Use/Improve  Data 

RAND  has  spent  decades  using  acquisition  data  to  solve  difficult  questions  on  a 
variety  of  defense  acquisition  topics.  Answering  sophisticated  acquisition  questions  requires 
analysts  with  detailed  knowledge,  access,  and  experience  with  numerous  data  sets.  They 
also  need  knowledge  of  how  the  information  systems  and  their  data  have  changed  over  time 
to  do  trend  and  other  analyses.  When  utilizing  very  large  data  sets,  robust  processing  and 
storage  capacity  and  the  skills  of  research  programmers  are  critical. 

The  DoD  needs  to  ensure  that  its  workforce  is  educated  and  trained  to  fully 
understand,  analyze,  and  use  existing  acquisition  data  opportunities.  The  acquisition 
community  must  have  the  skills  and  aptitude  to  understand,  analyze,  and  use  this  data  to 
make  decisions.  Lastly,  but  importantly,  the  DoD  needs  to  continue  to  focus  on  developing 
internal,  organic  capability  to  use  and  improve  acquisition  data  to  better  understand  what 
data  is  being  collected,  what  data  should  be  collected,  and  how  that  information  can  inform 
DoD  decision-making. 

References 

Data  element,  (n.d.).  In  PC  Magazine  online  encyclopedia.  Retrieved  March  30,  2016,  from 
http://www.pcmaq.com/encvclopedia/term/40771/data-element 

Kendall,  F.  (2013,  April  24).  Implementation  directive  for  Better  Buying  Power  2.0 — 
Achieving  greater  efficiency  and  productivity  in  defense  spending  [Memorandum]. 
Washington,  DC:  Office  of  the  Under  Secretary  of  Defense  (AT&L]). 

McKernan,  M.,  Riposo,  J.,  Drezner,  J.  A.,  McGovern,  G.,  Shontz,  D.,  &  Grammich,  C. 

(2016).  Issues  with  access  to  acquisition  data  and  information  in  the  Department  of 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-301  - 


Defense:  A  closer  look  at  the  origins  and  implementation  of  controlled  unclassified 
information  labels  and  security  policy  (RR-1476-OSD).  Santa  Monica,  CA:  RAND. 

Office  of  the  Secretary  of  Defense.  (2015,  January  7).  Operation  of  the  defense  acquisition 
system  (DoD  Instruction  5000.02).  Washington,  DC:  Author. 

Riposo,  J.,  McKernan,  M.,  Drezner,  J.  A.,  McGovern,  G.,  Tremblay,  D.,  Kumar,  J.,  & 
Sollinger,  J.  M.  (2015).  Issues  with  access  to  acquisition  data  and  information  in  the 
Department  of  Defense:  Policy  and  practice  (RR-880-OSD).  Santa  Monica,  CA:  RAND. 
Retrieved  from  http://www.rand.org/pubs/research  reports/RR880.html 

Structured  data.  (n.d.).  In  PC  Magazine  online  encyclopedia.  Retrieved  March  30,  2016, 
from  http://www.pcmaq.com/encvclopedia/term/52162/structured-data 

Unstructured  data.  (n.d.).  In  PC  Magazine  online  encyclopedia.  Retrieved  March  30,  2016, 
from  http://www.pcmaq.com/encvclopedia/term/53486/unstructured-data 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-302- 


Panel  9.  The  Operational  and  Developmental 
Dimensions  of  Cybersecurity 


Wednesday,  May  4,  2016 

3:30  p.m.  - 
5:00  p.m. 

Chair:  Rear  Admiral  David  H.  Lewis,  USN,  Commander,  Space  and  Naval 
Warfare  Systems  Command 

The  Cybersecurity  Challenge  in  Acquisition 

Sonia  Kaestner,  Adjunct  Professor,  McDonough  School  of  Business, 
Georgetown  University 

Craig  Arndt,  Professor,  Defense  Acquisition  University 

Robin  Dillon-Merrill,  Professor,  Georgetown  University 

Improving  Security  in  Software  Acquisition  and  Runtime  Integration  With 

Data  Retention  Specifications 

Daniel  Smullen,  Research  Assistant,  Carnegie  Mellon  University 

Travis  Breaux,  Assistant  Professor,  Carnegie  Mellon  University 

Cybersecurity  Figure  of  Merit 

CAPT  Brian  Erickson,  USN,  SPAWAR 

Rear  Admiral  David  H.  Lewis,  USN — was  born  at  Misawa  Air  Force  Base,  Japan,  and 
commissioned  in  1979  through  the  Navy  ROTC  Program  at  the  University  of  Nebraska.  As  the 
SPAWAR  Commander,  he  leads  a  global  workforce  of  9,700  civilian  and  military  personnel  who 
design,  develop,  and  deploy  advanced  communications  and  information  capabilities. 

At  sea,  Lewis  has  served  aboard  USS  Spruance  (DD  963)  as  communications  officer,  where  he 
earned  his  Surface  Warfare  qualification;  USS  Biddle  (CG  34)  as  fire  control  officer  and  missile 
battery  officer;  and  USS  Ticonderoga  (CG  47)  as  combat  systems  officer.  His  major  command 
assignment  was  Aegis  Shipbuilding  Program  manager  in  Program  Executive  Office  Ships,  where  he 
led  the  delivery  of  seven  DDG  51  class  ships  and  procured  another  10  ships. 

Lewis’s  shore  assignments  include  Assistant  Chief  of  Staff  for  Maintenance  and  Engineering, 
Commander,  Naval  Surface  Forces;  the  Navy  Secretariat  staff;  Naval  Sea  Systems  Command  staff; 
Aegis  Shipbuilding  Program  Office;  Supervisor  of  Shipbuilding,  Bath;  and  Readiness  Support  Group, 
San  Diego. 

Upon  selection  to  flag  rank  in  2009,  Lewis  served  as  Vice  Commander,  Naval  Sea  Systems 
Command,  and  then  served  four  years  as  Program  Executive  Officer,  Ships,  where  he  directed  the 
delivery  of  18  ships  and  procurement  of  another  51  ships. 

Lewis  holds  a  Bachelor  of  Science  degree  in  computer  science  from  the  University  of  Nebraska  and  a 
Master  of  Science  degree  in  computer  science  from  the  Naval  Postgraduate  School.  Lewis’s  personal 
awards  include  the  Legion  of  Merit,  Meritorious  Service  Medal,  Navy  and  Marine  Corps 
Commendation  Medal,  Navy  and  Marine  Corps  Achievement  Medal,  and  various  service  and  unit 
awards. 


mtSTANTIA  PER  SCIENTIAL 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-303- 


The  Cybersecurity  Challenge  in  Acquisition 


Sonia  Kaestner — is  an  Adjunct  Professor  at  the  McDonough  School  of  Business  at  Georgetown 
University.  She  has  recently  conducted  a  Strategy  and  Portfolio  Management  research  study  for  the 
Defense  Acquisition  University  and  developed  the  course  material,  exercises,  and  instructor  support 
material  for  a  new  course  offering  at  DAU  on  this  topic.  In  addition,  she  has  extensive  consulting 
experience  in  the  benchmarking  and  risk  analysis  of  capital  projects,  quantitative  and  qualitative 
analysis,  and  training  and  development  management. 

Craig  Arndt — is  the  Department  Chair  for  Engineering  and  Technology  and  a  Professor  of  Systems 
Engineering  at  the  Defense  Acquisition  University.  Dr.  Arndt  studies  the  development  of  new  methods 
for  the  development  of  taxonomies  of  learning  environments.  This  research  effort  is  developing 
methods  that  will  allow  future  educational  designers  to  select  and  develop  new  learning  environments 
based  on  the  nature  of  the  curriculum  and  the  requirements  and  demographics  of  the  student 
population.  He  holds  a  PhD  in  electrical  engineering. 

Robin  L.  Dillon-Merrill — is  a  Professor  in  the  McDonough  School  of  Business  at  Georgetown 
University.  She  seeks  to  understand  and  explain  how  and  why  people  make  the  decisions  that  they 
do  under  conditions  of  uncertainty  and  risk.  She  is  currently  in  year  two  of  a  multi-year  funded 
research  project  with  the  Department  of  Homeland  Security  titled  Beyond  Technical  Solutions  to 
Cybersecurity  Risk  Management  and  Risk  Communication:  Utilizing  Tools  and  Research  from 
Behavioral,  Economic,  and  Policy  Research. 

Abstract 

To  improve  cybersecurity,  the  acquisition  community  must  understand  and  manage  multiple 
dimensions  of  cyber-attacks  both  as  an  opportunity  and  as  a  risk  that  can  compromise  the 
bottom  line  of  the  organizations  they  work  for  and  with.  In  particular,  the  acquisition 
community  must  understand  and  recognize  the  cyber  threats  inherent  in  procuring  complex 
modern  systems  with  significant  cyber  components.  If  cybersecurity  is  not  designated  as  a 
requirement  of  a  modern  system,  it  is  often  challenging  to  add  effective  security  on  later,  and 
the  severity  of  the  cyber  vulnerabilities  may  only  be  identified  after  a  breach  has  already 
occurred.  If  appropriate  cybersecurity  is  designed  and  built-in,  these  systems  will  have  higher 
up-front  costs  but  potentially  lower  life-cycle  costs  because  of  the  reduced  need  to  fix 
vulnerabilities  in  the  systems  later.  Additionally,  individuals  working  in  acquisition  need  to 
recognize  that  given  the  sensitive  nature  of  their  work,  including  intellectual  property  and 
financial  data,  their  IT  processes,  information,  and  systems  will  be  an  attractive  target  for 
cyber  threats  from  both  criminal  sources  (e.g.,  organized  crime)  and  nation  state  adversaries, 
and  the  complexity  and  integration  of  the  modern  supply  chain  will  add  vulnerabilities  to  these 
linked  supplier  systems. 

Introduction 

Cyber  Threat  Challenges 

The  accelerated  growth  in  cyber/digital  technology  development  has  changed  the 
way  we  direct  our  lives,  business,  and  countries.  This  same  technology  development  has 
driven  the  rise  in  cybersecurity  breaches  through  the  increased  complexity  of  IT  systems, 
the  increased  use  of  personal  and  mobile  devices,  and  the  explosion  of  social  media.  In 
addition,  as  users,  we  have  not  had  the  same  speed  to  grow  the  skills  and  capabilities 
required  to  safely  absorb  the  technologies  we  now  depend  on.  So  far,  there  are  no 
cybersecurity  risk  management  readiness  standards,  and  organizations’  employees  (at  all 
levels)  lack  the  cybersecurity  training  required  to  prevent  and/or  promote  a  cyber-attack.  The 
lack  of  leadership’s  understanding  of  potential  vulnerabilities  and  liabilities  leads 
organizations  to  address  these  risks  mainly  from  a  technical  perspective,  and  hence,  rely 
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mainly  on  IT  professionals  to  solve  the  problem.  This  common  approach  ignores  the 
vulnerabilities  that  an  untrained  workforce  represents. 

According  to  the  UK  Information  Commissioner’s  Office,  93%  of  cybersecurity 
incidents  are  caused  by  human  error  (errors  even  when  designing  cybersecurity  processes 
and  systems);  thus  this  workforce  (untrained  and  sometimes  even  trained)  is  the  weakest 
link  in  the  cybersecurity  chain.  The  remaining  7%  was  due  to  technical  failures. 

The  consequences  of  cyber-attacks  are  diverse  and  include  the  suspension  of 
system  operations,  the  loss  of  current  and  future  revenue,  the  loss  of  intellectual  property, 
reputation  harm,  decreased  customer  confidence,  leaks  of  sensitive  information,  and  legal 
liability,  among  others.  These  consequences  are  often  exacerbated  because  attacks  are  not 
always  detected  immediately.  Verizon  (2013)  estimates  that  62%  of  data  breaches  were  not 
detected  for  at  least  several  months  if  not  longer. 

The  Identity  Theft  Resource  Center  (ITRC;  2016)  found  that  in  2015,  the 
Health/Medical,  Banking/Credit/Financial,  Government/Military  and  the  Education  sectors 
were  the  most  affected  by  cybercrime,  but  these  data  may  be  underestimating  the  scale  of 
the  cybercrime,  as  often  firms  (predominantly  small-  and  medium-size  business)  do  not 
disclose  cyber-attacks  to  attempt  to  avoid  the  financial  costs,  liability,  and  loss  of  goodwill 
that  come  with  disclosure  and  notification  (Supply  Chain  Quarterly,  2015). 

Some  progress  is  being  made  in  increasing  the  recognition  of  cybersecurity 
problems.  As  of  2015,  a  survey  conducted  by  PricewaterhouseCoopers  (PwC)  described 
that  over  two-thirds  of  organizations  were  more  concerned  about  cyber  threats  (PwC,  2015) 
than  in  previous  years’  studies.  Also,  the  U.S.  Department  of  Homeland  Security  (DHS)  has 
included  cybersecurity  as  a  top  priority  for  2016,  and  $587.5  million  have  been  allocated  to 
fund  programs  to  enhance  cybersecurity  situational  awareness  and  information  sharing 
(National  Cybersecurity  Protection  System  and  Continuous  Diagnostics  and  Mitigation; 

DHS,  2016).  Despite  all  the  media  coverage,  government  guidance,  and  the  increasing 
awareness,  cybersecurity  risk  management  continues  to  not  been  widely  implemented  or 
standardized  among  all  target  levels:  Individuals,  Organizations  and  Critical  Infrastructure. 
This  paper  will  begin  to  address  some  of  the  training  and  education  needed  to  improve 
cybersecurity  risk  management,  particularly  in  acquisition. 

Target  Levels:  Individuals 

As  shown  in  Figure  1 ,  individuals  face  the  risk  of  losing  data  and  privacy,  having  their 
devices  hacked  for  a  ransom  (denial  of  service — DoS),  or  simply  damaged.  In  the  first  week 
of  March  2016,  Mac  users  were  targeted  by  hackers  with  “ransomware”  in  what  is  believed 
to  be  the  first  complete  attack  campaign  of  its  kind  against  users  of  Apple’s  operating 
system.  Also,  several  incidents  have  been  reported  where  internet-enabled  baby  monitors 
have  been  hacked  to  disturb  infants.  This  last  example  depicts  the  increasing  risks 
associated  with  the  Internet  of  Things  (loT),  the  expanding  network  of  billions  of  everyday 
objects/devices  with  network  connectivity  and  data-sharing  capabilities  that  are  part  of  our 
daily  lives.  This  inappropriate  use  of  these  everyday  devices  was  certainly  not  considered 
when  they  were  designed.  How  individuals  use  pieces  of  an  acquired  system  and  where 
individual  personal  devices  plug  into  an  acquired  system  will  be  a  challenge  for  the 
acquisition  professional  to  understand  and  consider. 
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Figure  1 .  Target  Levels  for  Cyber-Attacks 
Target  Levels:  Organizations 

Organizations  face  the  same  risks  as  individuals,  but  to  a  larger  and  more  complex 
extent,  as  they  own  customer  and  proprietary  data  that  represent  the  core  of  their  business. 
In  addition,  they  also  face  legal  liability  from  their  customers,  reputation  damage,  and  higher 
exposure  to  the  loT  through  their  employers,  suppliers,  and  customers.  In  a  recent  example 
(February  2016),  the  Hollywood  Presbyterian  Medical  Center  was  victim  of  a  DoS  attack  that 
locked  employees  out  of  their  systems  by  encrypting  files  for  which  only  the  hackers  had  the 
decryption  key.  The  communications  between  physicians  and  medical  staff  were  paralyzed, 
and  they  suddenly  had  to  rely  on  paper  records  to  keep  operations  running.  The  hospital 
chose  to  pay  hackers  a  ransom  of  $17,000  in  bitcoins  to  regain  control  of  their  computer 
systems  after  the  cyber-attack.  The  origin  of  the  computer  network  intrusion  remains 
unknown  at  this  time.  The  ever  increasing  sophistication  of  the  cyber-attacks  makes  the  job 
of  the  acquisition  workforce  ever  more  challenging.  Following  the  Hollywood  Presbyterian 
Medical  Center  “ransomware”  attack,  this  is  now  a  new  concern  for  those  responsible  for  the 
acquisition  of  medical  record  systems.  Since  the  cyber  threats  keep  changing  and  evolving, 
how  is  a  manager  trained  in  acquisition  supposed  to  keep  track? 

Target  Levels:  Critical  Infrastructure 

Critical  Infrastructure  organizations  face  all  previously  mentioned  risks,  but  with 
bigger  consequences,  such  as  the  suspension/restriction  of  normal  operations  of  whole 
communities.  The  Ukrainian  power  grid  cyber-attack  in  2015,  for  example,  caused  a 
blackout  for  hundreds  of  thousands  of  people  in  Ukraine.  The  attack  used  destructive 
malware  that  wrecked  computers  and  wiped  out  sensitive  control  systems  for  parts  of  the 
Ukrainian  power  grid.  A  team  of  hackers  coordinated  attacks  at  the  same  time  against  six 
power  providers.  The  attack  was  so  severe  that  it  knocked  out  internal  systems  intended  to 
help  the  power  companies  restore  power.  Computers  were  destroyed,  and  even  the  call 
centers  used  to  report  outages  were  knocked  out.  The  source  of  the  attack  is  still  under 
investigation  (but  many  suspect  it  originated  in  Russia).  Since  the  risks  associated  with 
critical  infrastructure  are  so  great,  it  is  imperative  for  acquisition  specialists  in  these 
environments  to  understand  the  relevant  and  evolving  threats  to  their  computer  systems. 
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External  Attackers  Versus  Insiders  in  the  Supply  Chain 

Most  frequently,  cybersecurity  is  perceived  as  a  risk  from  the  outside  (i.e., 
hackers/criminals)  getting  illicit  access  to  the  organization’s  data/assets  with  ill  purposes. 
However,  organizations  are  not  adequately  addressing  the  internal  type  of  cyber  risk  that 
includes  employees  and  third-parties  which  have  access  to  critical  assets.  Among 
organizations  with  cybersecurity  risk  mitigation  plans,  48%  have  not  considered  third-party 
vendors,  43%  have  not  examined  the  role  of  contractors,  58%  have  not  examined  the  role  of 
suppliers,  and  92%  have  not  assessed  the  supply  chain  risk  management  as  a  whole  (PwC, 
2014,2015). 

External  types  of  cyber-attacks  against  prominent  organizations  such  as  JPMorgan 
Chase  and  the  U.S.  Central  Command  get  plenty  of  attention.  Cyber-attacks  involving 
internal  resources  (i.e.,  business  partners  and  direct  employees)  do  not  get  the  same 
coverage  despite  the  fact  that  they  pose  more  malicious  threats.  Internal  resources  have 
much  easier  access  to  systems  and  a  much  greater  window  of  opportunity. 

The  Target  and  Home  Depot  attacks  provide  good  examples  where  third-party 
contractors  unintentionally  facilitated  the  breach.  The  Target  breach  was  traced  back  to 
stolen  network  credentials  from  a  third  party  vendor  (a  refrigeration,  heating  and  air 
conditioning  subcontractor).  Home  Depot  reported  that  criminals  used  a  third-party  vendor’s 
user  name  and  password  to  enter  the  perimeter  of  their  network  and  then  acquired  elevated 
rights  that  allowed  them  to  navigate  portions  of  Home  Depot’s  network  and  to  deploy 
unique,  custom-built  malware  on  its  self-checkout  systems  in  the  United  States  and  Canada. 

Additionally,  at  a  general  level,  ill-trained  direct  employees  also  pose  significant 
insider  cybersecurity  risks.  Their  inability  to  identify  cyber  threats  leads  them  to 
unintentionally  click  on  phishing  email  links,  download  malware,  access  sensitive  data  from 
mobile  or  personal  devices,  etc.  At  a  more  specific  level,  professionals  who  are  in  charge  of 
designing  and  acquiring  products  and  systems  do  not  have  the  cybersecurity  knowledge  to 
identify  potential  risks  in  the  programs  they  are  designing,  managing,  or  acquiring.  Cyber 
risks  are  then  left  in  systems  during  the  requirements,  design,  and  contracting  of  the 
systems  development,  too  often  only  to  be  discovered  later  during  operations  after  an 
attack. 


It  is  widely  known  that  there  is  a  significant  shortage  of  cybersecurity  professionals. 
On  average  it  takes  three  months  to  hire  a  cybersecurity  professional,  as  only  25%  of  the 
applicants  meet  the  requirements  for  the  position,  but  over  70%  of  these  finally  hired 
professionals  lack  the  ability  to  understand  the  organization’s  business  (CSX,  2015).  There 
are  plenty  of  efforts  made  and  resources  allocated  to  close  the  cybersecurity  talent  gap. 
However,  the  focus  of  these  concerns  is  related  to  cyber  professionals  with  a  technical 
background  to  create  the  protection  walls  around  the  organization’s  assets,  and  to  create 
the  systems  to  detect  and  respond  to  threats.  This  type  of  professional  is  in  low  supply 
because  organizations  (of  all  sizes)  decided  in  the  early  2000s  to  send  “low-level”  IT  work, 
such  as  network  and  systems  administrators,  offshore  to  reduce  costs.  These  same 
organizations  missed  the  opportunity  to  grow  and  groom  those  professionals  that  they  need 
now.  The  solution  for  this  type  of  shortage  is  going  to  require  the  collaboration  of  universities 
(to  create  cyber-specific  careers  or  add  cybersecurity  training  to  complement  other  fields), 
marketing  campaigns  (increase  the  awareness  of  a  cyber  career  as  an  attractive  option), 
private  and  government  incentives  (e.g.,  scholarships),  and  so  forth. 

The  other,  and  mostly  ignored,  cybersecurity  talent  gap  is  related  to  the  employees 
at  the  different  levels  of  the  organization.  This  is  a  more  specific  talent  gap  that  requires  a 
customized  type  of  training  that  accounts  for  the  roles  each  employee  plays  and  the  type  of 
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data  he/she  has.  Certain  functions  need  to  have  an  extensive  cybersecurity  knowledge 
comparable  to  the  IT  professional  to  complement  their  non-IT  role.  For  example,  engineers 
who  are  designing  the  next  product  need  to  have  extensive  cybersecurity  knowledge  to 
close  potential  cybersecurity  gaps  in  their  designs  to  prevent  a  future  cyber  breach. 

The  Car  Manufacturing  Case 

As  can  be  seen  in  most  areas  of  technology,  computers  have  become  common 
components,  and  this  is  especially  true  in  the  cars  we  drive.  The  use  of  computers 
embedded  into  systems  does  not  in  itself  create  cybersecurity  vulnerabilities.  However, 
adding  computer-based  capabilities  to  existing  systems  creates  the  opportunities  for  a  wide 
range  of  cyber  risks  and  threats. 

The  hackers  are  publicizing  their  work  to  reveal  vulnerabilities  present  in  a  growing 
number  of  car  computers.  All  cars  and  trucks  contain  anywhere  from  20  to  70  computers. 
They  control  everything  from  the  brakes  to  acceleration  to  the  windows  and  are  connected 
to  an  internal  network.  A  few  hackers  have  recently  managed  to  find  their  way  into  these 
intricate  networks. 

In  one  case,  a  pair  of  hackers  manipulated  two  cars  by  plugging  a  laptop  into  a  port 
beneath  the  dashboard  where  mechanics  connect  their  computers  to  search  for  problems. 
Scarier  yet,  another  group  took  control  of  a  car’s  computers  through  a  cellular  telephone  and 
Bluetooth  connections  and  could  access  systems  including,  for  example,  the  tire  pressure 
monitoring  system. 

“The  more  technology  they  add  to  the  vehicle,  the  more  opportunities  there  are  for 
that  to  be  abused  for  nefarious  purposes,”  says  Rich  Mogull,  CEO  of  Phoenix-based 
Securosis,  a  security  research  firm.  “Anything  with  a  computer  chip  in  it  is  vulnerable,  history 
keeps  showing  us.” 

Two  years  ago,  researchers  at  the  University  of  Washington  and  University  of 
California,  San  Diego  did  more  extensive  work,  hacking  their  way  into  a  2009  midsize  car 
through  its  cellular,  Bluetooth,  and  other  wireless  connections.  Stefan  Savage,  a  UCSD 
computer  science  professor,  said  he  and  other  researchers  could  control  nearly  everything 
but  the  car’s  steering.  “We  could  have  turned  the  brakes  off.  We  could  have  killed  the 
engine.  We  could  have  engaged  the  brakes,”  he  said.  Savage  wouldn’t  identify  which 
manufacturer  made  the  car  they  hacked  into.  But  two  people  with  knowledge  of  the  work 
said  the  car  was  from  General  Motors  and  the  researchers  compromised  the  OnStar  safety 
system,  best  known  for  using  cellular  technology  to  check  on  customers  and  call  for  help  in 
a  crash.  The  people  didn’t  want  to  be  identified  because  they  were  not  authorized  to  speak 
publicly  on  the  matter  (“Hackers  Find  Weaknesses,”  2013). 

When  we  look  at  the  underlying  causes  of  the  current  generation  of  hacking  attacks 
on  the  auto  industry,  we  look  at  the  basics  of  the  mechanisms  of  cybersecurity.  First  is  the 
threat;  given  the  current  state  of  interest  in  the  hacker  community  and  the  ubiquitous  nature 
of  cars  in  the  United  States  there  will  be  a  continuing  and  rapidly  evolving  level  of  threat 
against  cars  now  that  there  is  an  understanding  that  accessing  their  networks  is  possible. 

Next  are  the  vulnerabilities.  There  are  a  number  of  different  vulnerabilities  that  could 
be  exploited  in  any  of  the  new  car  designs  as  has  been  noted  above.  As  we  look  for  lessons 
learned  to  build  better  systems  we  look  at  where  and  when  the  vulnerabilities  are 
introduced.  The  different  vulnerabilities  fall  into  three  principle  categories:  design 
vulnerabilities,  interface  vulnerabilities,  and  supply  chain  vulnerabilities.  In  our  case,  all  of 
their  vulnerabilities  were  introduced  by  the  people  in  the  car  companies  who  designed  the 
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car  and  did  not  anticipate  how  cyber  threats  could  work  because  that  was  not  their  job  to 
understand  the  technical  details  of  cyber  intrusion. 

Acquisition  professionals  need  to  have  the  technical  and  cybersecurity  skills  to 
identify  potential  gaps  in  the  products  and  systems  they  are  acquiring.  This  type  of 
cybersecurity  talent  gap  needs  to  be  addressed  through  the  implantations  of  training 
programs  customized  to  close  the  specific  talent  gaps  in  critical  functions  of  the 
organization. 

Cybersecurity  does  not  always  have  the  strategic  priority  it  should.  According  to  a 
Ponemon  Institute  (2015)  study,  as  of  2015,  only  34%  of  companies  consider  cybersecurity 
to  be  a  strategic  priority,  and  thus,  it  is  unlikely  that  enough  resources  are  allocated  to 
support  something  that  is  not  seen  as  a  priority.  Acquisition  organizations  have  to  assess 
their  acquisition  strategies  to  include  cyber  security  and  need  to  focus  mitigation  efforts  to 
include  all  parties  involved  in  the  organization’s  supply  chain. 

Acquiring  Cyber  Secure  Systems 

Understanding  and  recognizing  the  cyber  threats  inherent  in  procuring  complex 
modern  systems  with  significant  cyber  components  is  a  challenge.  For  example,  as  was 
mentioned  in  the  car  manufacturing  example,  in  201 1 ,  the  vulnerability  of  telematic  systems 
like  GM’s  OnStar  was  demonstrated  to  not  require  hacking  but  just  identification  of  each 
equipped  car’s  OnStar  telephone  number.  That  flaw  was  later  fixed,  but  highlights  the 
challenges  of  understanding  the  vulnerabilities  of  new,  complex  modern  networked  systems. 
As  described  by  Greenberg  (2015),  from  the  time  the  problem  was  first  identified  until  it  was 
fixed  was  more  than  five  years.  Greenberg  (2015)  goes  on  to  explain,  “Automakers  five 
years  ago  simply  weren’t  equipped  to  fix  hackable  bugs  in  their  vehicles’  software  ...  and 
many  of  those  companies  may  not  be  much  better  prepared  today.” 

Training  the  acquisition  workforce  to  understand  the  complex  cyber  challenges  of 
their  systems  at  the  right  level  of  detail  is  the  only  viable  solution  to  this  problem  in  the  long 
run. 

Securing  Supply  Chains 

Technology  development  has  significantly  changed  the  way  organization  conduct 
their  business: 


The  flexibility,  scalability,  and  efficiency  of  the  technology  that  enables 
information  sharing  among  partners,  has  created  additional  points  of  access 
to  an  organization’s  proprietary  information,  increasing  the  risks  that  the 
corporate  knowledge  that  drives  profitability  may  fall  into  the  wrong  hands. 
(Supply  Chain  Quarterly,  2015) 

Any  vendor  with  company  credential  access  can  expose  the  internal  network  to  an 

attack. 


As  shown  in  Figure  2,  the  acquisition  challenge  is  based  on  the  complexity  of  the 
supply  chain  that  most  organizations  have,  which  in  many  cases  includes  both  upstream 
(i.e.,  suppliers)  and  downstream  (i.e.,  market)  components  and  the  global  environment. 
More  and  more,  organizations  are  required  to  share  information  with  suppliers,  contractors, 
third-party  vendors  (and  their  vendors — fourth-party  partners),  that  do  not  have  the  same 
approach  to  cybersecurity.  Cybersecurity  vulnerabilities  in  their  supply  chains  will  in  turn 
introduce  new  vulnerabilities  in  the  organization  and  must  be  managed  by  those  acquisition 
specialists  focused  most  on  the  supply  chain  relationships. 
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Figure  2.  Supply  Chain  Complexity 

In  addition,  the  supply  chain  not  only  is  the  mechanism  that  develops  and  delivers 
products  and  services  from  source  to  customer,  but  also  represents  critical  parts  of  the  value 
chain  system  (inbound  logistics,  operations,  and  outbound  logistics)  in  which 
interdependence  is  the  fundamental  tenet  behind  gaining  a  competitive  advantage  (Porter, 
1985).  Organizations  share  proprietary  data  across  their  value  chain  (e.g.,  marketing,  dales, 
pricing,  metrics,  point-of-sale  information,  inventory  flows,  enterprise  system  activities,  etc.), 
increasing  the  number  of  potential  cyber  breach  entry  points. 

From  an  organization’s  strategic  point  of  view,  consolidating  the  supply  chain  is 
critical  to  reduce  costs  and  develop  integrated  profit  centers.  The  efficiency  of  the  global 
supply  chain  is  highly  dependent  on  the  speed  data  is  transferred  among  supply  change 
partners.  How  to  do  this  without  introducing  more  cybersecurity  risks  is  the  challenge 
organizations  have  now  to  address. 

Vertically  integrated  organization  (upstream  and  downstream  operations)  will  carry  a 
higher  risk  profile  than  a  horizontally  integrated  organization.  For  example,  in  the 
Volkswagen  (VW)  emissions  case,  the  organization  (OEM1)  was  able  to  install  deceiving 
software  to  cheat  on  the  emissions  testing  for  its  diesel  cars.  The  cars’  computers  were  able 
to  alter  how  their  engines  worked  to  reduce  emissions  (to  meet  required  levels  of  pollutants) 
while  they  were  being  tested.  Customers  became  aware  of  this  practice  (when  the  cars  have 
left  the  supply  chain)  after  six  years  of  “successful”  implementations  of  the  software. 

Although  in  this  case,  VW  was  fully  responsible  for  the  implementation  of  this  software. 

Using  the  same  approach,  cyber  criminals  could  use  the  same  strategy  to  benefit  from  the 
potential  damage  to  an  organization. 

Many  breaches  seen  so  far  have  been  because  of  a  lack  of  standardized 
credentialing  processes  and  a  lack  of  technology  updates  and  patches.  As  organizations 
share  their  information  with  their  business  partners  (through  the  internet,  mobile  devices, 


1  Original  Equipment  Manufacturer 
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cloud  computing,  etc.),  their  cybersecurity  vulnerability  increases,  opening  new  doors  for 
hackers  (Wall  Street  Journal  [WSJ],  2014). 

Third-  and  fourth-party  supplier’s  technology  use  also  represents  a  challenge  as 
organizations  do  not  have  control  over  the  type  of  technology  and  technology  upgrades 
those  parties  rely  on.  In  2015,  over  40%  of  large  and  medium  size  organizations  in  the 
United  States  and  UK  were  still  using  Windows  XP,  which  is  no  longer  supported  by 
Microsoft,  and,  hence,  no  up-to-date  security  upgrades  are  available  (Prince,  2015). 
According  to  Microsoft,  Windows  XP  users  are  five  times  more  vulnerable  to  security  risks 
and  viruses  than  organizations  using  up-to-date  operating  systems.  Based  on  these 
statistics,  most  likely  many  suppliers  are  still  using  Windows  XP. 

The  most  frequent  supply  chain  attacks  are  related  to  malware,2  compromised 
credentials,3  distributed  denial  of  services  (DDoS),4  and  SQL  injections.5  Supply  chain 
partner  relations  bring  an  additional  cybersecurity  potential  entry  point  (Supply  Chain 
Quarterly,  2015): 

•  Vendor  relationships  and  global  information  transmission 

•  Open  access  to  data  rather  than  “need  to  know”  access 

•  Frequent  changes  in  suppliers  and  products 

•  Lack  of  standardization  of  security  protocols  across  vendors  and  other 
partners 

•  Infected  devices  on  a  corporate  network 

•  Obsolete  security  infrastructure  or  outdated  hardware/software 

The  vulnerabilities  of  these  multiple  entry  points  need  to  be  recognized,  monitored, 
and  addressed  by  the  acquisition  specialists. 

Responsibility  and  Accountability 

Historically,  IT  managers  were  responsible  and  accountable  for  any  issues  related  to 
the  cyber  world  which  was  viewed  as  a  technology-centered  issue.  Almost  half  of  most 
organization  leadership  still  views  cybersecurity  risk  as  an  IT  matter,  rather  than  an 
organization-wide  risk.  Many  organizations  (46%;  PWC,  2014)  do  not  have  a  leadership  role 
such  as  a  Chief  Information  Security  Officer  (CISO)  or  a  Chief  Information  Officer  (CIO)  to 
centralize  all  cyber  related  issues. 

Supply  chain  cyber  risk  cannot  be  outsourced  and  can  only  be  address  with  a  holistic 
and  collaborative  risk  mitigation  plan  that  includes  effective  collaboration  of  a 
multidisciplinary  team  that  includes  not  only  IT  professionals  but  also  supply  chain,  finance, 
and  HR  professionals  and,  foremost,  the  support  of  the  senior  leadership  (PWC,  2015). 


2  Malicious  software  that  is  imbedded  on  computers,  devices,  or  networks,  damaging  files  (e.g., 
spyware,  worms,  viruses,  and  Trojan  horses) 

3  Unauthorized  use  of  usernames  and  passwords  to  access  a  company’s  network 

4  Disruption  systems  or  networks  to  prevent  the  normal  operations  of  the  organization 

5  Insertion  of  malicious  code  into  Structured  Query  Language  (SQL)  to  illegally  access  proprietary 
data,  bypassing  firewalls  and  other  security  measures 
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Creating  effective  cybersecurity  operations  requires  significant  in-house  resources,  but  at 
the  end  of  the  day,  it  is  the  only  way  to  protect  the  organization’s  data  (CFO,  2015). 

The  cybersecurity  approach  is  now  also  expanding  the  technology-centered  view  to 
include  people  and  processes. 

IT  leadership  is  required  to  manage  all  technical  aspects  required  to  address 
cybersecurity  risks  (both  hardware  and  software).  They  also  play  a  key  role  creating 
systems  and  processes  to  mitigate  the  risks  and  communicating  cyber  threats  to  the 
organization’s  highest  leadership  groups. 

Supply  chain  managers  need  to  understand  how  the  cybersecurity  risk  management 
process  of  their  suppliers  could  expose/affect  their  own  organization.  They  also  need  to 
understand  the  type  of  threats  the  organization  faces,  the  assets  that  are  under  risk,  and 
how  the  IT  department  is  handling  these  risks. 

Finance  leadership  support  is  required  not  only  to  support  the  cybersecurity  program 
among  the  organization,  but  also  to  identify  threats  and  quantify  the  financial  impact  of  cyber 
risk  (CFO,  2015). 

The  Human  Resources  (HR)  managers  also  play  a  key  role  to  prevent  the  hiring  and 
contracting  of  employees  who  pose  high  risk  (intentional  or  not)  to  the  organization. 
Research  has  shown  that  people  who  are  willing  to  conduct  or  assist  in  cyber-attacks  suffer 
from  one  or  more  identifiable  conditions  (Machiavelism,  narcissism,  and  psychopathy)  and 
have  a  combination  of  these  personality  traits:  immaturity,  low  self-esteem,  amorality  and 
lack  of  ethics,  superficiality,  lack  of  conscientiousness,  manipulativeness,  and  instability.  HR 
managers  can  look  out  for  threats  when  hiring  and  contracting. 

The  other  role  that  HR  plays  in  cybersecurity  is  in  the  recruiting  and  retention  of 
highly  skilled  cybersecurity  experts.  This  is  a  prevalent  challenge  for  organizations  of  all 
sizes.  Either  because  cybersecurity  employees  choose  to  create  their  own  security  firm  or 
they  move  to  a  more  attractive  job  position,  the  current  shortage  is  affecting  the  way 
organizations  deal  with  the  cyber  risk. 

Given  that  cybersecurity  breaches  carry  a  series  of  financial,  operational, 
reputational,  and  legal  damages,  senior  leadership  involvement  becomes  more  critical  to 
support  the  development  and  implementation  of  a  sound  cybersecurity  risk  mitigation 
program.  At  the  end  of  the  day,  the  magnitude  of  the  consequences  will  make  them 
accountable  for  the  approach  the  organization  has  taken  to  address  cyber  risks. 

Cost  &  Benefits  of  Cybersecurity 

As  serious  as  the  cybersecurity  risk  is,  it  does  not  receive  the  attention  and  priority  it 
requires  among  organizations  across  industries.  Most  cybersecurity  budgets  are  inadequate 
to  address  the  organization’s  risk  until  a  breach  becomes  a  reality.  A  sound  cybersecurity 
cost  benefit  analysis  is  usually  done  post  mortem  (after  an  attack)  and  then  generally  by  a 
third-party,  such  as  the  media.  The  Heartland  Payment  Systems  Inc.  attack  in  2015  (more 
than  100  million  credit  and  debit  card  numbers  were  stolen)  is  a  good  example  of  this 
analysis.  The  company  had  to  pay  $150  million  in  fines  and  legal  costs  and  suffered 
damage  to  its  reputation  as  a  payment  processor.  To  address  future  liabilities,  the  company 
quadrupled  its  security  budget,  reduced  the  number  of  computer  systems  that  process  credit 
and  debit  card  data,  and  added  more  encryption  and  system-monitoring  tools. 

The  WSJ  (2014)  describes  an  interesting  metric  tracked  by  Gartner  Inc.  which  states 
that  for  every  $5.62  a  business  spent  after  a  breach,  an  organization  could  spend  $1  before 
an  attack  on  encryption  and  network  protection  to  prevent  intrusions  and  minimize  damages. 
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As  obvious  as  this  might  look,  not  all  organizations  appear  to  justify  the  investment,  as  they 
assume  the  investment  cost  will  be  higher  than  the  cost  related  to  the  breach.  There  might 
be  different  reasons  behind  this  negligence,  and  we  discuss  four:  the  accounting 
perspective,  the  human  nature  perspective,  the  financial  perspective,  and  the  political 
perspective. 

Accounting  Perspective 

Since  cybersecurity  investments  do  not  generate  revenue,  it  is  usually  treated  as  an 
expense  of  doing  business  and  not  part  of  the  net  profit  of  the  organization.  Hence,  the 
potential  impact  is  not  proactively  and  consistently  quantified  (much  less  the  assessment  of 
the  intangible  consequences  of  a  cyber-attack,  such  as  the  credibility,  reputation,  legal 
expenses,  etc.).  But  even  so,  a  more  reasoned  approach,  as  Dew  Smith  (Supply  Chain 
Quarterly,  2015)  suggested,  would  be  to  revise  the  accounting  method  used  to  include  IT 
and  cybersecurity  spending  into  the  cost  methodology  of  supply  chain  management 
(absorption  costing6).  This  method  would  consider  cybersecurity  spending  as  part  of  the 
total  direct  cost  (including  overhead  cost  associated  with  logistics,  sales/marketing,  and 
manufacturing). 

Human  Nature  Perspective 

Behavioral  and  brain  science  research  shows  that  the  human  moral  judgement 
system  drives  the  urgent  need  for  actions  when  it  deems  the  issue  at  task  a  moral 
imperative  (the  principle  originating  inside  a  person’s  mind  that  compels  that  person  to  act). 
Reasons  that  cyber  risk  is  not  currently  registering  as  a  moral  imperative  could  be  (1)  that 
cyber  risk  is  communicated  as  an  abstract  and  complex  potential  event  (no  immediate  threat 
with  a  specific  shape),  and  hence  it  does  not  generate  a  rapid  emotional  intuitive  reaction; 

(2)  that  it  is  not  perceived  as  an  intentional  moral  transgression  on  the  part  of  employees, 
and  therefore  is  judged  less  severely  than  if  it  were  an  intentional  act;  and  (3)  it  is  deemed  to 
be  an  uncertain  event  (may  or  may  not  happen)  too  far  away  in  the  future  which  promotes 
unrealistic  optimisms  (it  will  not  happen  to  us),  and  this  optimism  prevents  people  from 
identifying  themselves  as  a  target.  Changing  the  way  in  which  organizations  communicate 
cybersecurity  risk  can  change  the  way  we  perceive  its  urgency  to  act. 

Financial  Perspective 

Driven  by  a  financial  statement/budget  compliance  focus,  some  may  argue  that  for 
some  organizations  (especially  large  ones),  the  losses  involved  are  so  small  compared  to 
their  revenue  that  it  is  easier  to  take  a  chance  and  write  off  any  losses  should  they  occur. 

For  example,  Target’s  data  breach  had  a  $252  million  cost  during  2013  and  2014.  After 
insurance  coverage  and  tax  deductions,  Target  ended  up  paying  $105  million,  which  is 
about  0.1%  of  its  2014  revenue.  Similarly,  Home  Depot  paid  $28  million,  after  the  $15  million 
insurance  payment,  which  represents  0.01%  of  the  company’s  revenue  the  same  year  (CBS 
News,  2015).  This  approach  not  only  does  not  assess  the  full  consequences  of  a  data 
breach  (e.g.,  competitive  advantage,  brand  equity,  customer  loyalty,  reputation,  etc.),  but 
also  ignores  the  ethical  component.  The  responsibility  to  protect  customers’  data,  inform 
them  of  the  breach,  and  gain  back  their  trust  still  lies  with  the  organization,  and  it  is  not 
reflected  in  the  financial  statements. 


6  All  of  the  manufacturing  costs  are  absorbed  by  the  units  produced. 
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Since  201 1 ,  the  SEC7  has  urged  organizations  to  provide  details  about  the 
operational  and  financial  risk  posed  by  cyber-attacks  in  the  risk  section  of  their  filings  and 
discuss  with  the  investors  (in  the  Management’s  Discussion  and  Analysis  section)  any 
effects  of  cyber-attacks  on  operating  results,  liquidity,  or  financial  position.  So  far,  investors 
have  not  been  satisfied  with  the  information  provided  and  feel  the  disclosures  were 
presented  “merely  for  legal  prophylaxis,  instead  of  for  informing  investors”  (Fortune,  2015). 

Political  Perspective 

In  2014,  the  Obama  administration  issued  new  cybersecurity  guidelines  urging 
companies  in  critical  infrastructure  industries  to  increase  their  efforts  to  protect  and  monitor 
their  networks  and  train  employees.  Some  organizations  took  the  guidelines  as  non¬ 
commercial  because  cybersecurity  measures  must  be  cost-effective  for  an  individual 
company  or  be  supported  by  some  economic  incentives.  Some  suggested  that  if  the 
government  wants  to  improve  cyber-defense,  the  government  should  subsidize  the  cost 
(e.g.,  tax  breaks).  To  complicate  this  matter,  many  regulators  consider  this  problem  more  of 
a  corporate  responsibility  that  national  security  (CFO,  2105). 

Organizations  perform  cost-benefit  analysis  for  expenses.  The  cybersecurity  cost 
(i.e.,  budget  allocation)  then  represents  a  careful  balancing  act  where  it  is  critical  to  identify 
the  right  amount  of  security  given  the  risks  the  organization  is  exposed  to. 

Trends 

Cybersecurity  Approach 

Cybersecurity  risk  management  has  been  focused  on  preventing  cybercrime  through 
the  use  of  internal  controls,  employee  training,  and  firewalls,  among  others.  Acknowledging 
that  it  is  impossible  to  protect  a  network  against  100%  of  the  attacks,  it  is  key  to  include  a 
plan  to  address  the  possible  breaches  to  minimize  the  damage.  According  to  Heather 
Crofford,  CFO  of  Shared  Services  at  Northrop  Grumman,  “detections,  response  and 
recovery  are  where  the  increasing  investment  needs  to  be”  (CFO,  2015). 

Supply  Chain  Analytics  (Souza,  2014),  Cyber  Risk  Modeling  (CFO,  2015),  &  Big 

Data  (PwC,  2016) 

Supply  chain  analytics  currently  focuses  on  the  use  of  information  and  analytical 
tools  to  make  better  decisions  regarding  the  material  flows  in  the  supply  chain.  Some  of 
these  same  concepts  and  tools  can  also  be  used  for  cybersecurity  purposes  (including  the 
supply  chain  cyber  risk).  The  availability  of  Big  Data  and  the  use  of  descriptive  and 
predictive  analytics  could  prove  useful  as  tools  to  fight  cybersecurity  threats. 

Descriptive  analytics8  tools  can  be  quite  useful  to  provide  a  clear  view  of  the  current 
situation  of  the  suppliers.  Supply  chain  mapping  is  an  example  where  an  organization  can 
map  all  their  suppliers  (and  their  suppliers)  and  plot  them  using  different  criteria  such  as  the 
importance  in  the  organization’s  supply  chain,  level  of  cybersecurity  maturity,  etc.  Figure  3 
illustrates  how  the  French  Nuclear  Power  Supply  Chain  is  mapped  using  one  of  the  currently 
available  tools  (Sourcemap,  n.d.). 


7  Security  and  Exchange  Commission 

8  Uses  existing  information  to  evaluate  what  is  happening 
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Figure  3.  Example  of  Supply  Chain  Mapping  Using  Sourcemap.com 

Predictive  analytics  can  use  past  data  to  forecast  future  cyber  risks,  including  those 
coming  from  the  supply  chain.  Data  source  and  quality  are  important  components  to  use 
these  tools  (e.g.,  linear  and  non-linear  regression  and  data  mining). 

Despite  the  limited  detail  data  availability  (both  the  causes  of  the  breach  and  balance 
sheet  impact),  some  insurance-related  organizations  are  already  focusing  their  efforts  on 
using  data  analytics  in  cyber  risk  modeling  to  help  assess  their  clients’  cyber  risk.  In  theory, 
the  frequency  of  cyber-attacks  is  rapidly  increasing  the  amount  of  data  to  be  analyzed. 
However,  the  number  of  organizations  who  are  cyber-attack  victims  who  are  willing  to  share 
these  types  of  data  is  pretty  small,  and  hence  current  models  are  based  only  on  publicly 
available  data  from  various  insurance  sources. 

Furthermore,  in  2015,  almost  60%  of  private  and  government  organizations  used  Big 
Data  analytics  to  model  and  monitor  cybersecurity  threats,  respond  to  incidents  and  audit 
and  review  data  to  understand  how  it  is  used,  by  whom,  and  when.  As  more  data  become 
available,  this  trend  is  expected  to  significantly  increase  in  the  next  few  years. 

Cloud  Enabled  Cybersecurity,  Advanced  Authentication  (PwC,  2016) 

Cloud  service  providers  have  invested  significant  amounts  in  advanced  technologies 
for  data  protection,  privacy,  network  security,  and  identity  and  access  management.  The 
most  frequently  used  cloud-based  cybersecurity  services  include  real-time  monitoring  and 
analytics,  advanced  authentication,  identity  and  access  management,  threat  intelligence, 
and  end-point  protection. 

Simple  password  use  is  no  longer  an  adequate  way  to  access  data.  All  industries  are 
quickly  migrating  to  the  use  of  advanced  authentication  to  help  manage  access  and  improve 
trust  among  customers  and  business  partners.  Combinations  of  one-time  passwords  and 
hardware  tokens,  biometrics,  security  keys,  and  special  applications  are  the  most  common 
advanced  authentication  methods  used. 

Cybersecurity  Risk  Management  Practices 

Risk-Based  Cybersecurity  Frameworks 

Most  private  and  government  organizations  use  a  standard  framework,  or  a 
combination  of  multiple  frameworks,  currently  available  to  develop  an  effective  cybersecurity 
program.  The  most  frequently  used  are  the  National  Institute  of  Standards  and  Technology 
(NIST)  framework  and  ISO  27001  (Information  security  management).  In  addition,  there  are 
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other,  more  acquisition  specific  frameworks,  such  as  the  Centre  for  the  Protection  of 
National  Infrastructure  (CPNI)  Framework  (methodology)  which  helps  develop  supply  chain 
specific  security  risk  mitigation  implementation  plans,  ISO  2800-2700  (Specification  for 
security  management  systems  for  the  supply  chain),  and  the  Supplier  Assurance  Framework 
(UK  Cabinet  Office,  2015). 

The  NIST  Framework  was  developed  after  President  Obama’s  executive  order  on 
“Improving  Critical  Infrastructure  Cybersecurity”  and  is  quickly  becoming  a  standard  among 
industries  in  the  United  States.  The  Framework  consolidates  existing  global  standards  and 
practices  to  help  organizations  understand,  communicate,  and  manage  their  cyber  risks 
(White  House,  2014).  The  Framework  offers  a  road  map  to  develop  a  cybersecurity  program 
for  organizations  with  no  security  experience.  For  organizations  with  a  more  mature 
cybersecurity,  the  Framework  helps  them  improve  the  communication  with  the 
organization’s  leadership  and  suppliers  about  cyber  risk  management.  The  Framework  has 
three  components:  Core,  Tiers,  and  Profile. 

The  Framework  core  is  a  set  of  cybersecurity  activities,  desired  outcomes,  and 
applicable  references  that  are  common  across  critical  infrastructure  sectors.  The  Framework 
has  four  implementation  Tiers  (Partial,  Risked  Informed,  Repeatable,  and  Adaptive)  to 
reflect  how  the  organization  views  cybersecurity  risk  and  assess  the  processes  in  place  to 
manage  that  risk.  The  Framework  profile  represents  the  outcomes  based  on  business  needs 
that  an  organization  has  selected  from  the  Framework  categories  and  subcategories. 

ISO  27001  was  developed  to  “provide  a  model  for  establishing,  implementing, 
operating,  monitoring,  reviewing,  maintaining  and  improving  an  information  security 
management  system.”  It  uses  a  top-down,  risk-based  approach  and  is  technology-neutral. 
Also,  it  provides  requirements  to  develop  an  information  security  management  system  to 
managing  sensitive  company  information  so  that  it  remains  secure.  It  includes  people, 
processes,  and  IT  systems  by  applying  a  risk  management  process.  It  can  help  small, 
medium,  and  large  businesses  in  any  sector  keep  information  assets  secure. 

The  whole  ISO  27000  family  aims  to  help  organizations  keep  information  assets 
(e.g.,  financial  information,  intellectual  property,  employee  details  or  information  entrusted  to 
you  by  third  parties)  secured. 

The  CPNI  proposes  that  supply  chain  security  risk  be  an  extension  of  existing  risk 
management  processes.  The  extensions  should  include: 

•  Comprehensive  maps  of  all  tiers  of  the  upstream  and  downstream  supply 
chains  to  the  level  of  individual  contracts 

•  Risk  scoring  each  contractor  to  link  in  to  the  organization’s  existing  security 
risk  assessment 

•  Due  diligence/accreditation/assurance  of  suppliers  (and  potential  suppliers) 
and  the  adoption,  through  contracts,  of  proportionate  and  appropriate 
measures  to  mitigate  risk 

•  Audit  arrangements  and  compliance  monitoring 

•  Contract  exit  arrangement 

ISO  2800-2007  (Specification  for  security  management  systems  for  the  supply  chain) 
offers  a  framework  for  providing  effective  physical  security  management  through  a  system 
that  identifies  security  threats,  assesses  risk,  establishes  objectives  for  implementing 
controls,  and  continuously  improves  the  physical  security  of  the  organization.  It  identifies 
requirements  for  implementing  and  operating  a  security  management  system,  including 
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organizational  (security)  structure,  authorized  personnel  responsible  for  security 
management,  assessing  and  maintaining  competence  of  personnel,  and  training  for 
personnel  responsible  for  security. 

The  Supplier  Assurance  Framework  (UK  Cabinet  Office,  2015)  applies  to  official 
contracts  and  enables  the  early  identification  of  high  risk  projects;  it  provides  a  framework 
for  the  risk  management  of  contracts  that  is  consistent,  effective,  and  understood  by 
government,  stakeholders,  and  suppliers  and  enables  information  sharing  and 
accountability.  It  is  flexible  enough  to  allow  its  customization  to  meet  specific  business 
needs.  It  is  particularly  relevant  where  information  is  shared  through  contracts  or 
agreements. 

To  different  extents,  all  these  frameworks  address  training  needs.  Note,  however, 
that  none  of  them  do  so  specifically  for  the  acquisition  community  and  much  less  to  the 
detail  it  is  required. 

High-Reliability  Organizations  (HROs) — An  Alternative  Approach  to  Cybersecurity 

As  previously  discussed,  cyber-attacks  are  mostly  driven  by  network  administrators 
and  users’  errors  rather  than  by  inadequate  security  technology.  Organizations  can 
implement  key  concepts  of  HROs  (Weick  &  Sutcliffe,  2015)  to  address  the  human  error 
component  of  cybersecurity  risks  as  the  U.S.  military  has  successfully  done.  The  basic 
principle  is  to  treat  the  unknown  as  knowable  by  following  some  key  principles: 

•  Mindful  organizing  (organizing  is  about  coordination) 

•  Preoccupation  with  failure 

•  Reluctance  to  simplify 

•  Sensitive  to  operations 

•  Commitment  to  resilience 

•  Deference  to  expertise 

The  U.S.  Navy’s  nuclear-propulsion  program  is  arguably  the  HRO  with  the  longest 
track  record.  There  are  six  principles  that  helped  the  Navy  contain  the  impact  of  human 
error:  (1)  integrity,  (2)  depth  of  knowledge,  (3)  procedural  compliance,  (4)  forceful  backups, 
(5)  a  question  of  attitude,  and  (6)  formality  in  communication. 

Building  an  HRO  requires  the  personal  attention  of  senior  leadership  as  well  as  a 
substantial  financial  investment  in  training  and  oversight.  This  is  approach  has  proven  to  be 
effective  at  the  whole  organization  level  and  can  certainly  be  extended  to  include  the 
acquisition  group  of  the  organization. 

Recommendations 

Considering  that  each  organization  has  a  different  cybersecurity  maturity  level,  the 
following  recommendations  are  directed  to  the  risks  on  the  acquisition  process,  and  hence, 
assume  there  is  already  a  risk  analysis  based  cybersecurity  risk  management  program  in 
place. 

Purely  technical  solutions  will  not  address  the  magnitude  of  the  risk.  Even  the  best 
technology  will  not  work  well  with  poorly  trained  operators.  Processes  and  people  need  to 
be  part  of  the  solution  in  order  to  deliver  a  comprehensive  cybersecurity  approach 
customized  to  address  the  cyber  risks  associated  to  the  supply  chain. 

To  improve  cybersecurity,  the  acquisition  community  must  understand 
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•  and  manage  the  multiple  dimensions  of  cyber-attacks  (opportunities  and 
risks)  that  can  compromise  the  bottom-line  of  the  organizations  they  work  for 
and  with. 

•  and  recognize  the  cyber  threats  inherent  in  procuring  complex,  modern 
systems  with  significant  cyber  components  and  the  challenges  of 
understanding  the  vulnerabilities  of  new,  complex  modern  networked 
systems. 

•  that  purchasing  products  and  services  that  have  the  appropriate 
cybersecurity  designed  and  built-in  may  have  higher  up-front  costs  but  lower 
life-cycle  costs  because  of  the  reduced  need  to  fix  vulnerabilities  in  the 
systems  later. 

•  that  given  the  sensitive  nature  of  their  work,  including  intellectual  property 
and  financial  data,  their  IT  processes,  information,  and  systems  will  be  an 
attractive  target  for  cyber  threats  from  both  criminal  sources  and  nation  state 
adversaries. 


Risk  Assessment 

Risk  management  experts  agree  that  the  first  step  to  take  is  to  assess  the  financial 
risk  of  a  security  breach.  This  requires  a  detailed  inventory  of  the  organization’s  assets  at 
risk  that  will  be  used  to  assess  the  financial  risk.  However,  the  training  of  the  professionals 
who  will  be  assessing  the  cyber  risks  should  be  a  step  even  before  this,  as  the  validity  of  the 
cyber  risk  assessment  will  be  as  good  as  the  cyber  risk  skill  and  knowledge  the  employees 
who  perform  the  analysis  have. 

Subsequently,  organizations  need  a  detailed  accounting  of  all  firms  (partners, 
affiliates,  network  participants,  etc.)  that  are  part  of  the  supply  chain  (both  upstream  and 
downstream)  to  identify  the  weakest  link  then,  assess  the  degree  of  reliance  of  each  of 
those  organizations  (size  and  scope). 

Finally,  survey  and  audit  all  third-party  partners’  (i.e.,  fourth-party  contractors) 
cybersecurity  process/programs  and  capabilities  to  identify  the  level  of  risk  each  of  them 
carry.  All  this  information  will  allow  the  organization  to  create  a  vendor  compliance  protocol 
and  strategic  outsourcing  guidelines  to  ensure  a  standard  level  of  cybersecurity  across  the 
supply  chain. 

New  vendor  compliance  can  be  achieved  through  the  consistent  implementation  of 
cybersecurity  incentives/requirements.  This  can  include  but  is  not  limited  to  the  requirement 
of  cybersecurity  protocols,  conditions,  and  capabilities  to  be  aligned  with  the  organization’s 
cybersecurity  risk  mitigation  process  as  part  of  the  contract  approval  criteria. 

The  case  is  slightly  different  with  existing  vendors,  as  contracts  have  already  been 
awarded.  In  this  case,  a  contract  amendment  (allowed  within  the  law)  to  include  the  new 
cybersecurity  requirements  is  the  easiest  way.  In  the  event  this  is  not  feasible,  the 
procurement  and  IT  groups  should  create  a  process  to  mitigate  the  risks  those  existing 
vendors  bring  to  the  organization. 

The  greater  the  complexity  of  the  supply  chain,  the  more  extensive  the  risk 
management  efforts  should  be.  Therefore,  organizations  with  a  complex  supply  chain  should 
include  multiple  layers  of  security  (e.g.,  redundant  backup  systems,  multiple-stage  access 
thresholds  for  credentials,  ongoing  threat  monitoring,  etc.). 
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Insurance  Use  for  Commercially  Developed  Systems 

To  reduce  the  financial  impact  of  a  breach,  organizations  are  including  cyber  liability 
insurances  in  their  organizations’  plans.  Cyber  insurance  organizations  provide  partial 
protection  against  internet-based  risks  relating  to  information  technology  infrastructure  and 
information  assets.  Typical  first-party  coverage  includes  forensic  investigation,  legal  advice, 
notification  costs  of  communicating  the  breach,  credit  monitoring,  PR  expenses,  loss  of 
profits  and  extra  expenses  during  the  time  your  network  is  down.  Common  third-party 
coverage  includes  legal  defense,  settlements,  damages  and  judgements  related  to  the 
breach,  liability  to  banks  for  re-issuing  credit  cards,  cost  of  responding  to  regulatory  inquires, 
and  regulatory  fines  and  penalties  (WS&Co,  2014). 

The  use  of  cyber  insurance  is  meaningful  for  large  organizations  with  auditable 
cybersecurity  programs.  However,  some  small-  or  medium-size  organizations  might  get  a 
false  sense  of  security  from  cyber  insurance  and  fail  to  implement  a  sound  cybersecurity 
program  (Market  Watch,  2015). 

From  an  acquisition  perspective,  suppliers  who  have  cyber  insurance  might  indicate 
a  higher  level  of  cyber  maturity  as  insurance  companies  perform  extensive  cyber  audits 
before  securing  a  policy. 

Implementation  of  KPIs  to  Monitor  Progress 

Having  a  cybersecurity  program  that  includes  supplier  risk  is  not  enough  to  conclude 
the  threats  are  under  control.  The  performance  of  this  program  needs  to  be  continuously 
monitored  to  address  the  dynamic  nature  of  the  risks.  The  use  of  Key  Performance 
Indicators  (KPIs)  has  been  proven  as  an  effective  way  to  communicate  challenges  and 
opportunities.  The  cyber  world  includes  technological,  process/procedural,  and  people  KPIs 
that  can  be  implemented  to  assess  the  effectiveness  of  the  current  cybersecurity  program 
(Dowdy,  Hubback,  &  Solyom,  2014).  KPIs  can  also  be  used  to  assess  the  level  of  risk  each 
supplier  brings  to  the  organization. 

Technological  KPIs  focus  on  the  number  and  type  of  electronic  touchpoints  and 
highlight  the  quality  of  management  of  these  connections.  An  example  of  such  a  KPI  is  the 
number  of  days  that  elapse  between  Microsoft  issuing  a  critical  software  update  and  the 
entire  organization  installing  it.  Process/procedural  KPIs  can  include  data-policy  and 
operational  policy  indicators  to  assess,  for  example,  if  the  percent  of  sensitive  data 
encryption  meets  the  current  policies,  or  if  the  number  of  attempted  security-policy  breaches 
within  a  certain  period  meets  industry  standards.  People  KPIs  (including  business  partners’ 
employees)  measure  the  success  rate  of  training,  employee  conformity  to  security 
guidelines,  or  employee  knowledge  and  use  of  best-practice  e-mail  behavior.  They  may  be 
assessed  through  spot  tests. 

It  is  certainly  a  good  practice  to  include  cybersecurity  metrics  into  the  organizations 
KPIs,  balance  scorecard  and/or  executive  dashboard.  Given  the  current  technology  focus  of 
cybersecurity,  it  will  require  the  effective  training  of  both  cybersecurity  professionals  and 
employees  to  identify  (and  implement)  a  set  of  meaningful  KPIs  that  will  bridge  technology 
issues  with  business  context  to  better  respond  to  the  needs  of  the  organization. 

The  Future 

As  of  2015,  only  34%  of  U.S.  organizations  (30%  UK/Europe  and  28%  Middle 
East/North  Africa)  were  prepared  to  deal  with  cybersecurity  risks  resulting  from  the  loT 
(Ponemon  Institute,  2015).  Current  predictions  assess  the  number  of  devices  connected  to 
the  internet  will  reach  30  billion  by  2020  (IDC,  2015).  From  2014  to  2015,  the  number  of 
incidents  related  to  the  loT  has  increased  by  over  150%.  Most  of  the  loT  attacks  were 
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related  to  mobile  devices,  embedded  systems,  consumer  technologies  and  operational 
systems  (PwC,  2016).  The  obvious  consequence  of  the  loT  is  that  the  cyber  risk  penetration 
area  will  increase  in  size  and  complexity.  Organizations  need  to  consider  a  strategy  to  deal 
with  risks  created  by  the  internet  of  things. 

Early  in  2015,  President  Obama  signed  the  “Promoting  Private  Sector  Cybersecurity 
Information  Sharing”  executive  order  to  enable  private  and  government  organizations  to 
share  industry  specific  information  and  intelligence  related  to  geographies,  issues,  events, 
or  specific  threats  through  the  creation  of  new  Information  Sharing  and  Analysis 
Organizations  (ISAOs).  The  goal  of  these  ISAOs  is  to  address  cyber  threats  to  public  health 
and  safety,  national  security,  and  economic  security  of  the  United  States  by  sharing 
information  related  to  cybersecurity  risks  and  incidents  from  private  companies,  nonprofit 
organizations,  executive  departments  and  agencies  (agencies),  and  other  entities. 
Organizations  need  to  join  ISAOs  or  similar  organizations  to  share  and  receive  cyber 
intelligence. 
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Abstract 

The  Department  of  Defense  (DoD)  Risk  Management  Framework  (RMF)  for  IT  systems  is 
aligned  with  the  National  Institute  for  Standards  and  Technology  (NIST)  guidance  for  federal 
IT  architectures,  including  emergent  mobile  and  cloud-based  platforms.  This  guidance  serves 
as  a  prescriptive  lifecycle  for  IT  engineers  to  recognize,  understand,  and  mitigate  security 
risks.  However,  integrators  are  left  with  the  challenge — during  acquisition  and  during  runtime 
integration  with  external  services — to  reason  about  the  actions  on  data  inherent  in  their 
system  designs  that  may  have  confidentiality  risks.  These  risks  may  lead  to  data  spills,  loss 
of  confidentiality  for  mission  data,  and/or  revelations  about  private  data  related  to  service 
members  and  their  families.  Solutions  are  needed  to  assist  acquisition  professionals  to  align 
system  data  practices  with  the  RMF  and  NIST  guidance,  as  well  as  DoD  IA  directives — 
particularly  with  respect  to  the  collection,  usage,  transfer,  and  retention  of  data.  To  provide 
support  to  this  end,  we  extended  our  initial  automation  framework  to  support  reasoning  over 
data  retention  actions  using  a  formal  language.  We  propose  an  evaluation  method  for  these 
extensions,  carried  out  through  simulations  of  real-world  IT  systems  using  imitation  but 
statistically  accurate  synthetic  data.  Our  language  aims  to  address  dynamically  composable, 
multi-party  systems  that  preserve  security  properties  and  address  incipient  data  privacy 
concerns.  Software  developers  and  certification  authorities  can  use  these  profiles  expressed 
in  first-order  logic  with  an  inference  engine  to  advance  the  RMF,  express  data  retention 
actions  that  promote  confidentiality,  and  re-evaluate  risk  mitigation  and  compliance  as  IT 
systems  evolve  over  time. 
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Cybersecurity  Figure  of  Merit 


CAPT  Brian  Erickson,  USN— SPAWAR 
Executive  Summary 

Over  time,  Navy  warfighters  have  grown  accustomed  to  the  promise  of  reliable 
information  technology,  based  on  the  experience  of  more  than  30  years  of  relative  peace  in 
this  domain.  However,  inspections,  Red  Teams,  and  actual  adversaries  have  shown  that 
this  reliance  on  interconnected  technology  is  not  well-founded.  In  2012,  the  Defense 
Science  Board  determined  that 

nearly  every  conceivable  component  within  DoD  is  networked.  These 
networked  systems  and  components  are  inextricably  linked  to  the 
Department’s  ability  to  project  military  force  and  the  associated  mission 
assurance.  Yet,  DoD’s  networks  are  built  on  inherently  insecure  architectures 
that  are  composed  of,  and  increasingly  using,  foreign  parts.  While  DoD  takes 
great  care  to  secure  the  use  and  operation  of  the  “hardware”  of  its  weapon 
systems,  the  same  level  of  resource  and  attention  is  not  spent  on  the 
complex  network  of  information  technology  (IT)  systems  that  are  used  to 
support  and  operate  those  weapons  or  critical  IT  capabilities  embedded 
within  them. 

In  many  cases,  the  Navy  bases  its  strategic  and  operational  plans  on  the  very 
capabilities  a  cyber  enemy  will  deny  or  exploit  in  war.  In  an  interview  with  CHIPS  (2014) 
magazine,  Matt  Swartz,  Director,  Communications  and  Networks  Division  Deputy  Chief  of 
Naval  Operations  for  Information  Dominance  (N2/N6)  and  Navy  Task  Force  Cyber 
Awakening  (TFCA)  Lead  said, 

Recent  real  world  events  and  attacks  on  our  Navy  systems  make  clear  the 
cyber  threat  is  increasing.  The  risk  calculus  in  the  cyber  domain  has 
changed.  Our  reliance  on  connected  capabilities  has  significantly  increased 
the  potential  consequences  of  a  cyber-attack. 

In  short,  the  Department  of  Defense  (DoD)  has  awakened  to  a  “reliance  vs. 
reliability”  gap  in  its  networks  that  the  services  struggle  to  measure. 

Within  the  Navy,  quality  efforts  to  achieve  and  measure  cybersecurity  across  multiple 
strategic,  operational,  and  tactical  level  commands,  program  offices,  and  systems  and  type 
commands  are  often  coordinated  primarily  through  assumed  adherence  to  overarching 
strategic  guidance.  No  single  organization  has  the  broad  area  visibility,  resources,  and 
influence  to  orchestrate  these  efforts. 

Compounding  the  problem  is  the  DoD’s  lack  of  a  consistent,  widely  accepted 
methodology  to  measure  and  clearly  and  concisely  express  the  operational  readiness  and 
programmatic  wholeness  of  Information  Technology-based  Programs  of  Record.  Existing 
acquisition  metrics  do  not  respond  to  current  operational  imperatives  and  cannot  adequately 
express  future  risk  to  mission.  The  Navy  is  unable  to  measure  and  express  cyber  program  of 
record  wholeness,  platform  cyber  readiness,  and  the  impact  of  programmatic  and  budgetary 
decisions  on  cyber  readiness,  or  to  quantify  the  value  of  specific  cybersecurity  standards  or 
controls.  Without  an  accepted  means  of  holistically  scoring  risk  within  a  system  of  systems 
construct,  the  Navy  cannot  consistently  shape  cybersecurity  investment  priorities  to  optimize 
value  in  a  resource  constrained  environment.  As  a  result,  the  Navy  struggles  to  effectively 
manage,  prioritize,  and  influence  the  allocation  of  resources  among  competing  systems 
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commands  and  program  offices  to  maximize  cyber  readiness  of  the  Fleet  and  to  defend 
those  decisions  in  the  planning,  programming,  and  budgeting  process. 

Cybersecurity  Figure  of  Merit 

This  paper  addresses  the  lack  of  a  consistent,  widely  accepted  means  of  measuring 
current  and  future  cyber  risk  to  mission  resulting  from  acquisition  or  operational  weaknesses 
in  cybersecurity  within  an  Information  Technology-based  Program  of  Record  through  the 
concept  of  a  Cybersecurity  Figure  of  Merit  (CFOM).  The  objective  is  to  develop  a 
transparent  mathematical  framework  of  weighted  qualitative  and  quantitative  metrics  that 
expresses  the  relative  effectiveness  of  an  Information  Technology-based  Program  of 
Record  in  terms  of  the  completeness  and  sufficiency  of  its  cybersecurity  properties 
throughout  its  lifecycle.  CFOM  can  be  used  to  address  acquisition  readiness  for  the 
Milestone  Decision  Authority  as  well  as  impacts  of  budget  decisions  on  the  cybersecurity 
wholeness  of  a  given  program. 

CFOM  is  developed  in  terms  of  operational  risk  as  a  function  of  threats, 
vulnerabilities,  and  consequences  (Defense  Science  Board,  2012).  While  CFOM  is  initially 
and  primarily  intended  as  an  acquisition  readiness  tool,  it  groups  acquisition  metrics  into 
operational  categories  to  maintain  acquisition’s  focus  on  providing  future  operational 
capability.  As  an  expression  of  operational  risk,  CFOM  predicts  future  operational  risk  by 
measuring  a  program’s  long-range  budget  and  spend  plan,  technology  refresh  plans, 
architecture,  and  sustainment  plans  through  various  technical  reviews  and  operational 
reports. 

CFOM  is  intended  to  be  a  practical  application  of  a  large  body  of  theoretical  and 
practical  research  available  on  metrics  and  scoring  of  cybersecurity.  As  such,  while  the 
authors  recognize  the  mathematical  tenets  of  probability  and  risk  theory,  choices  between 
practicality  and  academic  rigor  are  made  in  favor  of  practicality.  One  of  the  self-imposed 
study  restraints  for  initial  implementation  is  that  the  program  must  not  require  new 
inspections  or  changes  to  existing  documents  and  reports  to  gather  the  metrics  that  make 
up  CFOM. 

In  conclusion,  the  CFOM  framework  attempts  to  provide  decision  support  to  both 
acquisition  and  operations  by  enabling  greater  understanding  of 

•  a  program’s  acquisition  readiness  in  terms  of  future  operational  risk 
throughout  its  lifecycle. 

•  the  effect  of  today’s  budget  or  technology  tradeoffs  on  future  operational  risk. 

•  how  technical  risk  decisions  in  one  part  of  an  enterprise  network  translate  to 
operational  risk  at  the  tactical  or  operational  edge  elsewhere. 

•  how  to  optimize  complex  cybersecurity  investment  combinations  to  provide 
the  maximum  value  in  terms  of  operational  risk  reduction  in  resource- 
constrained  environments. 

Additional  work  is  needed  to  refine  the  framework  based  on  real-world  data  and  then 
to  materialize  the  capability  in  a  software  prototype. 
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advocates  for  financial  adjustments  to  optimize  force  readiness.  She  oversees  preparation  and 
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the  Naval  Supply  Systems  Command.  In  these  positions,  she  has  developed  logistics  programs  for 
the  Department  of  Defense,  implemented  one  of  the  first  integrated  and  agile  data-driven 
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Abstract 

As  the  current  budget  drawdown  has  progressed,  numerous  policy  makers  and  informed 
observers  have  expressed  concerns  about  the  effect  on  federal  research  and  development 
(R&D)  efforts.  Across  the  federal  government,  but  particularly  within  the  Department  of 
Defense  (DoD),  there  have  been  fears  that  the  sharp  downturn  in  federal  contract  obligations 
would  disproportionately  impact  the  R&D  contracting  portfolios  within  individual  agencies  and 
their  major  components. 

Looking  at  the  period  from  2000-2014,  this  report  examines  data  for  the  four  major  R&D 
contracting  agencies:  the  DoD,  NASA,  the  HHS,  and  the  Department  of  Energy.  It  also 
examines  four  hypotheses,  generated  by  the  study  team  from  a  review  of  the  literature  and 
consultation  with  experts,  that  test  how  the  budget  drawdown  has  affected  the  R&D 
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contracting  portfolios,  and  the  industrial  base  that  supports  those  efforts,  within  each  R&D 
contracting  agency. 

The  main  finding  of  this  initial  inquiry  is  that  the  conventional  wisdom  regarding  how  R&D 
contracting  would  be  affected  by  the  budget  drawdown  has  not  been  borne  out.  Early  stage, 
“seed  corn”  R&D  has  been  relatively  protected,  cuts  were  not  done  within  agencies  on  a 
“salami  slice”  basis,  and  large  prime  vendors  have  seen  their  shares  of  the  federal  R&D 
contracting  market  decline  precipitously. 

Introduction 

As  the  current  budget  drawdown,  resulting  from  fiscal  restraints  imposed  by  the 
Budget  Control  Act,  as  well  as  sequestration  and  its  aftermath,  has  progressed,  numerous 
policy  makers  and  informed  observers  have  expressed  concerns  about  the  effect  on  federal 
research  and  development  (R&D)  efforts.  Across  the  federal  government,  but  particularly 
within  the  Department  of  Defense  (DoD),  there  have  been  fears  that  the  sharp  downturn  in 
federal  contract  obligations  would  disproportionately  impact  the  R&D  contracting  portfolios 
within  individual  agencies  and  their  major  components.  Using  data  from  the  publicly- 
available  Federal  Procurement  Data  Systems  (FPDS),  this  report  examines  trends  with 
federal  R&D  contracting  during  the  current  drawdown  and  analyzes  the  degree  to  which 
actual  data  conforms  to  predicted  trends. 

In  order  to  analyze  trends  within  the  R&D  contracting  portfolios  of  the  four  largest 
federal  R&D  customers  (the  DoD,  Department  of  Energy  [DoE],  National  Aeronautics  and 
Space  Administration  (NASA),  and  Department  of  Health  &  Human  Services  [HHS]),  CSIS 
has  developed  a  methodology  to  categorize  R&D  contracts  by  stage  of  R&D  using  a 
categorization  schema  that  roughly  corresponds  to  the  commonly-used  DoD  R&D  Budget 
Activity  Codes  (BACs):1 

•  Basic  Research  (6.1) 

•  Applied  Research  (6.2) 

•  Advanced  Technology  Development  (ATD)  (6.3) 

•  Advanced  Component  Development  &  Prototypes  (ACD&P;  6.4) 

•  System  Development  &  Demonstration  (SD&D;  6.5) 

•  Operational  Systems  Development  (6.7) 

•  Operation  of  Government  R&D  Facilities  (GOCO)2 

The  following  section  (Federal  R&D  Contracting  in  Context)  looks  at  the  overall 
trends  for  federal  R&D,  both  by  which  federal  agency  or  major  component  is  doing  the 
contracting  and  by  what  stage  of  R&D  the  work  falls  under.  The  next  section  after  that — How 
Has  the  Budget  Drawdown  Affected  Federal  R&D  Contracting? — examines  four  hypotheses 
regarding  how  federal  R&D  will  be  affected  by  the  budget  drawdown,  drawn  from  the 
literature  and  from  consultation  with  experts,  and  examines  how  well  the  data  conforms  to 
those  predictions. 


1  CSIS  does  not  include  contracts  for  R&D  Management  Support  (6.6)  in  this  analysis. 

2  Though  not  classified  as  R&D  in  the  FPDS,  CSIS  now  includes  the  codes  for  management/ 
operation  of  federal  R&D  facilities  in  its  R&D  category,  as  a  significant  amount  of  R&D  activity, 
particularly  in  the  DoE,  is  structured  in  this  manner. 
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Based  on  the  available  data,  the  study  concludes  that  most  of  the  assumptions  that 
the  study  team,  policy  makers,  and  outside  experts  made  about  the  impact  of  the  budget 
drawdown  on  federal  R&D  contracting  were  not  borne  out.  While  R&D  contracting  portfolios 
in  some  parts  of  the  federal  government  saw  dramatic  cuts,  others  were  relatively 
preserved,  and  the  distribution  of  those  cuts  did  not  conform  to  expectations. 

Federal  R&D  Contracting  in  Context 

Four  federal  agencies  have  accounted  for  95%  or  more  of  total  federal  R&D  contract 
obligations  in  every  year  since  2000:  the  DoD,  the  DoE,  NASA,  and  the  HHS.  Of  these,  the 
DoD  accounts  for  by  far  the  largest  share,  with  over  50%  in  every  year  during  the  2000- 
2014  period,  reaching  as  high  as  66%  in  2007.  The  DoE  accounted  for  39%  of  total  federal 
R&D  contract  obligations  in  2000,  but  has  accounted  for  between  20%  and  25%  in  most 
years  since  2004.  NASA,  which  accounted  for  between  4%  and  5%  of  federal  R&D  contract 
obligations  from  2000-2003,  has  seen  steady  growth  since  then  and  has  accounted  for 
double-digit  shares  in  every  year  since  2009.  Meanwhile,  the  HHS  has  accounted  for 
between  3%  and  5%  of  total  federal  R&D  contract  obligations  in  all  but  one  year  in  the 
2000-2014  period  (6%  in  2013). 

Figure  1  shows  overall  federal  R&D  contract  obligations,  broken  down  by  customer, 
with  the  federal-wide  total  for  each  year  at  the  top  of  each  column. 
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Figure  1.  Federal  R&D  Contract  Obligations  by  Customer,  2000-2014 

(Federal  Procurement  Data  Systems  [FPDS];  CSIS  analysis) 

Since  their  peak  in  2009,  as  overall  federal  contract  obligations  declined  by  26%, 
federal  R&D  contract  obligations  have  declined  by  31%.  Interestingly,  most  of  this 
disproportionate  decline  in  federal  R&D  contracts  occurred  prior  to  the  impact  of 
sequestration — since  2012,  as  overall  federal  contract  obligations  declined  by  16%,  federal 
R&D  contract  obligations  fell  roughly  in  parallel  (-17%),  with  similarly  parallel  declines  in  both 
2013  and  2014. 
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To  better  understand  trends  within  the  federal  R&D  contracting  portfolio,  CSIS  has 
created  a  methodology  to  classify  R&D  contracts  by  stage  of  R&D,  using  the  widely- 
understood  Budget  Activity  Codes  (BACs)  as  a  guide.  Figure  2  shows  federal  R&D  contract 
obligations  by  stage  of  R&D. 
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Figure  2.  Federal  R&D  Contract  Obligations  by  Stage  of  R&D,  2000-2014 

(FPDS;  CSIS  analysis) 

As  overall  federal  R&D  contract  obligations  declined  by  31%  since  2009,  Basic 
Research  (-22%),  Applied  Research  (-12%),  ACD&P  (-22%),  Operational  Systems 
Development  (-12%),  and  GOCO  (-16%)  were  all  relatively  preserved.  Meanwhile,  ATD  (- 
47%)  and  SD&D  (-59%)  both  saw  dramatic,  disproportionate  declines.  As  a  share  of  overall 
federal  R&D  contract  obligations,  Basic  Research  and  Applied  Research,  combined,  rose 
from  30%  in  2009  to  36%  in  2014.  Meanwhile,  ATD  fell  from  17%  to  13%,  and  SD&D 
declined  from  21%  in  2009  to  13%  in  2014.  Overall,  the  current  drawdown  has  seen  a 
notable  shift  within  the  federal  R&D  contracting  portfolio,  with  a  greater  share  of  obligations 
going  to  early  stage,  “seed  corn”  R&D.  The  drivers  of  this  trend  will  be  analyzed  in  the 
sections  that  follow,  which  will  look  at  the  R&D  contracting  portfolios  within  the  major  federal 
R&D  customers. 

Department  of  Defense3 

Since  2009,  DoD  R&D  contract  obligations  have  declined  by  43%,  notably  faster 
than  the  31%  decline  in  overall  DoD  contract  obligations  over  this  same  period.  As  a  share 
of  overall  DoD  contract  obligations,  R&D  declined  from  11%  in  2009  to  9%  in  2014,  the 
lowest  share  seen  in  the  2000-2014  period. 


3  Portions  of  this  section  are  adapted  from  CSIS’  January  2016  report  on  overall  Defense  Acquisition 
Trends,  which  drew  in  part  upon  research  and  analysis  done  in  preparation  for  this  research  effort. 
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Throughout  the  budget  drawdown,  numerous  policy  makers  have  expressed  concern 
that  the  DoD  would  end  up  sacrificing  investment  in  “seed  corn”  R&D  in  order  to  preserve 
funding  for  later  stage,  more  established  development  programs.  But  as  Figure  3  shows, 
that  has  not  been  the  case. 
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Figure  3.  DoD  R&D  Contract  Obligations  by  Stage  of  R&D,  2000-2014 

(FPDS;  CSIS  analysis) 

Since  2009,  as  overall  DoD  R&D  contract  obligations  declined  by  43%,  obligations 
for  Applied  Research  declined  by  just  over  one-third  that  rate  (-15%), 4  while  obligations  for 
Basic  Research  declined  by  only  32%.  As  a  share  of  DoD  R&D  contract  obligations,  the  two 
“seed  corn”  categories  rose  from  27%  in  2009  to  38%  in  2014,  the  highest  share  in  the 
2000-2014  period.  Basic  Research  contract  obligations  have  declined  at  a  rate  that  more 
closely  parallels  the  overall  decline  in  DoD  R&D  contract  obligations  since  2012,  but  Applied 
Research  obligations  have  continued  to  be  relatively  preserved  (-18%  decline  since  2012, 
compared  to  -27%  for  overall  DoD  R&D.) 

Contract  obligations  for  ACD&P  (-23%)  and  Operational  Systems  Development  (- 
25%)  have  similarly  been  relatively  preserved  since  2009.  But  ATD  (-50%)  and  SD&D  (- 
66%)  have  seen  massive  declines  in  recent  years.  The  declines  in  those  two  stages  of  R&D 
accounted  for  over  three-quarters  of  the  total  decline  in  DoD  R&D  contract  obligations 
during  the  current  drawdown. 

The  enormous  decline  in  SD&D  is  particularly  telling  and  speaks  to  the  larger  trend  in 
DoD  R&D  contracting — over  the  last  several  years,  as  R&D  programs  related  to  MDAPs 


4  DoD  contract  obligations  for  Applied  Research  actually  saw  a  notable  spike  between  2009  and 
201 1 ,  due  primarily  to  a  one-year  spike  for  space-related  R&D,  but  obligations  returned  to  prior  levels 
in  2012. 


PBAlSTANTlA  PER  SCIENTIAL 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-332  - 


have  either  been  canceled  or  matured  into  production,  the  DoD  has  been  largely  unable  to 
start  and  sustain  new  development  programs,  either  due  to  budgetary  pressures  or  to 
programmatic  difficulties.  The  decline  in  R&D  contract  obligations  during  the  budget 
drawdown  is  being  driven  by  a  five-year  trough  in  the  pipeline  of  new  major  weapons 
systems. 

The  following  sections  will  briefly  examine  trends  in  R&D  contracting  within  the  three 
military  services. 

Army 

The  key  factor  in  the  massive  decline  in  Army  R&D  contract  obligations  (-61%  since 
2009,  compared  to  -52%  for  Army  contracts  overall)  has  been  the  cancellation  of  the  Army’s 
Future  Combat  Systems  (FCS)  program.  Nearly  the  entirety  of  the  decline  in  Army  R&D 
contract  obligations  between  2009  and  2012  is  directly  attributable  to  the  cancellation  and 
winding-down  of  the  FCS.  In  particular,  obligations  for  SD&D  have  declined  by  an  incredible 
94%  since  2009  as  the  Army  has  struggled  to  start  and  sustain  new  development  programs 
for  major  weapons  systems  in  the  wake  of  the  FCS’s  cancellation.  The  result  of  these 
struggles  is  the  current  five-year  trough  in  the  Army’s  development  pipeline  for  major 
weapons  systems. 

In  terms  of  “seed  corn”  R&D,  the  trend  within  the  Army  is  mixed.  Both  Basic 
Research  (-45%)  and  Applied  Research  (-49%)  have  been  relatively  preserved  since  2009. 
While  Basic  Research  fell  more  slowly  than  overall  R&D  throughout  the  period,  Army 
obligations  for  Applied  Research  actually  grew  between  2009  and  201 1 ,  before  declining  by 
nearly  half  in  2013.  In  2014,  combined  obligations  for  the  two  “seed  corn”  categories  are  at 
their  lowest  level  ($1 .6  billion)  in  the  2000-2014  period. 

This  interruption  of  the  developmental  pipeline  for  new  major  weapons  systems 
presents  an  unusual  opportunity  for  the  DoD  and,  particularly,  for  the  Army.  As  spending  on 
war  materiel  continues  to  be  replaced  by  funding  for  next-generation  priorities,  the  Army  has 
little  to  no  developmental  money  already  committed  to  projects.  Thus,  the  Army  has  an 
opportunity  to  take  a  step  back,  draw  lessons  from  the  wars  in  Iraq  and  Afghanistan, 
evaluate  potential  future  threats  and  missions,  and  direct  their  requirements  and 
developmental  priorities  accordingly. 

Navy 

While  overall  Navy  contract  obligations  were  relatively  preserved  (-19%)  since  2009, 
Navy  R&D  contract  obligations  fell  by  47%  over  that  same  period.  As  a  share  of  overall  Navy 
contract  obligations,  R&D  fell  from  14%  in  2009  to  9%  in  2014,  the  lowest  share  for  the  Navy 
in  the  2000-2014  period. 

Whereas  obligations  for  Advanced  Research  have  increased  by  3%  over  the  2009- 
2014  period,  obligations  for  Basic  Research  have  declined  by  two-thirds  since  2009.  As  with 
the  Army,  the  Navy  saw  disproportionate  declines  in  obligations  for  ATD  (-68%)  and  SD&D 
(-53%).  Unlike  the  Army,  the  Navy  has  major  development  programs  in  the  pipeline,  such  as 
the  Ohio-class  ballistic  missile  submarine  replacement.  However,  to  preserve  funding  for 
current  priorities,  the  Navy  has  been  forced  to  push  back  the  timelines  for  some  of  its  efforts 
due  to  budgetary  constraints,  resulting  in  the  ongoing  trough  in  the  Navy’s  development 
pipeline. 
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Air  Force 


As  with  the  Navy,  while  overall  Air  Force  contract  obligations  were  relatively 
preserved  (-24%)  since  2009,  R&D  contract  obligations  within  the  Air  Force  declined  more 
steeply  (-37%)  over  that  same  period.  Analogous  to  the  Army  and  Navy,  Air  Force  contract 
obligations  for  Applied  Research  were  relatively  preserved  since  2009  (-3%);  unlike  the 
Navy,  Basic  Research  was  also  relatively  preserved  (-25%)  and  actually  increased  by  11% 
in  2014.  As  a  share  of  Air  Force  R&D  contract  obligations,  “seed  corn  R&D”  rose  from  41% 
in  2009  to  58%  in  2014,  the  highest  share  in  the  2000-2014  period. 

Both  ATD  (-64%)  and  SD&D  (-58%)  declined  heavily,  with  most  of  the  declines 
coming  in  the  wake  of  sequestration  between  2012  and  2013.  Unlike  both  the  Army  and 
Navy,  however,  Air  Force  contract  obligations  for  ACD&P  also  declined  heavily  (-60%)  since 
2009.  The  Air  Force  is  also  in  the  midst  of  a  trough  in  their  development  pipeline  for  new 
major  weapons  systems,  but  with  contracts  recently  awarded  for  major  programs  like  the 
Long  Range  Strike  Bomber,  the  Air  Force  seems  like  it  will  be  the  first  of  the  military  services 
to  emerge  from  this  trough. 

NASA 

NASA’s  R&D  contract  portfolio  is  the  most  comparable  with  the  DoD’s  in  terms  of  the 
types  of  projects  undertaken,  if  not  in  overall  scale.  Basic  Research  and  Applied  Research 
have  combined  to  account  for  over  half  of  NASA  contract  obligations  in  all  but  one  year  in 
the  2000-2014  period,  peaking  at  73%  in  2005.  In  recent  years,  Applied  Research  has 
accounted  for  around  40%  of  overall  NASA  R&D  contract  obligations,  with  ATD  and  ACD&P 
declining  as  a  share  as  SD&D  obligations  grew.  Figure  4  shows  NASA  R&D  contract 
obligations  by  stage  of  R&D. 
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Figure  4.  NASA  R&D  Contract  Obligations  by  Stage  of  R&D,  2000-201 45 

(FPDS;  CSIS  analysis) 

Unlike  the  DoD,  NASA  R&D  contract  obligations  have  risen  steadily  since  2007,  with 
the  most  significant  growth  occurring  between  2007  and  2009,  primarily  in  ATD.  NASA  R&D 
contract  obligations  grew  by  9%  between  2009  and  2012,  fell  by  6%  in  2013,  and  by  a 
further  1%  in  2014;  for  the  entire  2009-2014  period,  NASA  contract  obligations  grew  by  3%, 
even  as  overall  NASA  contract  obligations  fell  by  10%.  Since  2012,  R&D  has  accounted  for 
over  half  of  NASA  contract  obligations,  the  highest  shares  (excluding  the  anomalous  2004) 
in  the  2000-2014  period. 

The  increase  in  R&D  contract  obligations  within  NASA  since  2009  has  been  driven 
by  significant  increases  in  Basic  Research  (74%)  and  SD&D  (69%),  while  obligations  for 
Applied  Research  (-9%),  ATD  (-33%),  and  ACD&P  (-24%)  declined  notably. 

Department  of  Health  and  Human  Services 

The  R&D  contracting  portfolio  of  the  HHS  has  diversified  notably  in  recent  years.  In 
2000,  Basic  Research  and  Applied  Research  combined  to  account  for  86%  of  HHS  R&D 
contract  obligations;  by  2014,  that  share  had  declined  to  57%.  Obligations  for  the  two 
categories  of  “seed  corn”  R&D  have  both  been  relatively  stable  over  the  2000-2014  period; 


5  The  $8.7  billion  in  2004  for  “Operation  of  Government  R&D  Facilities”  is  a  data  anomaly  related  to 
NASA’s  migration  from  their  previous  contract  data  system  into  FPDS.  In  the  prior  system,  large, 
multiyear  contracts  were  entered  as  a  single  aggregated  entry  at  the  end  of  the  contract;  this  entry 
represents  the  prior  five  years  of  obligations  for  NASA’s  contract  with  the  Jet  Propulsion  Lab  (JPL). 
CSIS  has  worked  with  NASA  contract  officials  at  the  JPL  and  has  determined  that  while  separating 
out  the  aggregated  total  is  not  feasible,  the  $8.7  billion  will  be  moved  back  to  2003,  which  was  the  last 
year  of  the  contract.  CSIS  would  like  to  thank  the  contract  officials  at  NASA  HQ  and  at  the  JPL  for 
their  diligence  and  assistance  in  tracking  down  this  data  anomaly. 
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the  decline  in  share  is  primarily  the  result  of  increasing  obligations  for  ATD  and  “Operation 
of  Government  R&D  Facilities”  starting  in  the  mid-to-late  2000s.  As  a  share  of  overall  HHS 
contract  obligations,  R&D  has  declined  steadily  throughout  the  period,  from  a  high  of  26%  in 
2004  to  13%  in  2014.  Figure  5  shows  HHS  R&D  contract  obligations  by  stage  of  R&D. 
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Figure  5.  HHS  R&D  Contract  Obligations  by  Stage  of  R&D,  2000-2014 

(FPDS;  CSIS  analysis) 

Since  2009,  as  overall  HHS  contract  obligations  fell  by  3%,  HHS  R&D  contract 
obligations  fell  by  29%,  albeit  after  a  33%  increase  between  2008  and  2009.  Basic 
Research  declined  by  42%  between  2009  and  2014,  but  that  was  primarily  the  result  of  a 
return  to  normal  obligation  levels  after  a  one-year  spike  in  2009;  between  2012  and  2014,  as 
overall  HHS  R&D  contract  obligations  were  virtually  flat,  obligations  for  Basic  Research 
increased  by  16%.  Obligations  for  Applied  Research  were  relatively  preserved  (-10%),  while 
obligations  for  ATD  increased  by  13%.  ATD  obligations  were  notably  volatile  during  this 
period,  doubling  between  2010  and  2011,  falling  by  a  third  in  2012,  increasing  by  144%  in 
2013,  and  then  falling  by  45%  in  2014. 

Department  of  Energy 

The  DoE  is  unique  among  the  major  federal  R&D  contracting  agencies  in  that  only  a 
small  percentage  of  its  R&D  contracting  portfolio  actually  goes  to  direct  contracts  for  R&D. 
Rather,  the  vast  majority  of  the  DoE’s  R&D  contract  obligations  go  to  “Operation  of  Federal 
R&D  Facilities,”  mainly  the  various  National  Laboratories.  Because  of  the  nature  of  these 
contracts,  CSIS  has  limited  visibility  to  the  nature  of  the  R&D  being  performed,  although 
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conversations  with  experts  have  indicated  that  most  of  the  R&D  activity  in  the  National 
Laboratories  would  probably  be  categorized  as  Basic  Research  or  Applied  Research.6 

The  DoE’s  R&D  contracting  portfolio  is  also  unique  in  that  almost  all  of  the 
obligations  in  recent  years  are  under  contracts  that  originated  in  2008  or  earlier.  In  2014,  for 
example,  less  than  2%  of  the  $14.5  billion  in  DoE  R&D  contract  obligations  came  from 
contracts  signed  after  2008,  and  35%  came  from  contracts  that  originated  in  2000  or  earlier. 
As  such,  DoE  R&D  contracting  data  has  limited  explanatory  value  regarding  the  effects  of 
the  current  drawdown,  since  almost  all  of  the  obligations  in  recent  years  come  from  options 
being  exercised  under  contracts  that  originated  before  the  drawdown  began. 

How  Has  the  Budget  Drawdown  Affected  Federal  R&D  Contracting? 

As  part  of  this  research  effort,  CSIS  has  conducted  a  review  of  the  relevant  literature, 
involving  both  the  public  and  private  sectors,  to  identify  current  theories  on  how  declining 
resources  would  affect  R&D  contract  spending.  CSIS  also  consulted  with  experts  in  federal 
contracting  and  budgeting  to  validate  the  theories  identified  in  the  course  of  the  literature 
review.  From  this  analysis,  the  study  team  developed  a  number  of  hypotheses  regarding 
how  declining  resources  would  affect  federal  R&D  contracting  overall,  and  the  R&D 
contracting  portfolios  within  agencies  specifically. 

This  section  looks  at  a  selection  of  these  identified  hypotheses  and  evaluates 
whether  the  predictions  made  by  the  study  team  (based  on  the  current  understanding  of  the 
issue  from  the  available  literature)  were  borne  out  by  the  data  on  the  current  budget 
drawdown.  The  following  are  the  five  hypotheses  that  this  section  will  examine: 

1 .  Cuts  in  R&D  due  to  budget  drawdown  will  be  done  on  a  “salami  slice”  basis, 
rather  than  reflecting  a  thoughtful  prioritization  of  resources. 

2.  Newer  R&D  contracts  will  bear  a  disproportionate  share  of  cuts  during  budget 
drawdowns. 

3.  Budget  drawdowns  will  lead  to  shifts  away  from  early-stage,  “seed  corn”  R&D 
towards  mid-to-late-stage  R&D  tied  to  high-profile  programs. 

4.  Large  prime  vendors  will  account  for  increasing  shares  of  federal  R&D  during 
budget  drawdowns. 

5.  During  budget  drawdowns,  R&D  will  be  increasingly  funded  out  of  non-R&D- 
focused  budget  accounts. 

Hypothesis  1:  Cuts  in  R&D  Due  to  Budget  Drawdown  Will  Be  Done  on  a  “Salami 
Slice”  Basis,  Rather  Than  Reflecting  a  Thoughtful  Prioritization  of  Resources. 

For  the  purposes  of  this  hypothesis,  the  study  team  uses  the  term  “salami  slice”  to 
refer  to  a  series  of  cuts  where  a  roughly  equal  portion  is  cut  across  the  board,  rather  than 
having  some  portions  of  the  portfolio  relatively  preserved  or  impacted.  Given  that 
sequestration,  in  particular,  was  implemented  as  an  across-the-board  cut,  CSIS 
hypothesized  that  agencies  would  respond  to  budgetary  pressures  by  taking  roughly  equal 


6  The  DoE  totals  for  “Operation  of  Federal  R&D  Facilities”  also  likely  include  some  production  activity 
related  to  nuclear  weapons,  but  CSIS  has  no  way  to  reliably  separate  these  out  from  the  R&D  activity 
undertaken  as  part  of  these  contracts.  As  such,  for  the  purposes  of  this  analysis,  CSIS  will  categorize 
the  “Operation  of  Federal  R&D  Facilities”  obligations  in  their  entirety  as  R&D. 
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cuts  across  their  R&D  contracting  portfolios.  If  this  hypothesis  were  to  hold  true,  the  study 
team  would  expect  to  find  that  across  the  different  stages  of  R&D  and  within  different  major 
components,  cuts  to  R&D  were  roughly  in  parallel  to  the  overall  decline  over  the  period,  if 
not  necessarily  in  each  particular  year. 

Department  of  Defense 

The  overall  DoD  R&D  contracting  portfolio  did  not  show  evidence  that  cuts  were 
done  on  a  “salami  slice”  basis.  While  Basic  Research  has  declined  roughly  in  parallel  to 
overall  DoD  R&D  contract  obligations  since  2012,  Applied  Research,  ACD&P,  and 
Operational  Systems  Development  have  all  declined  notably  more  slowly  than  overall  DoD 
R&D.  At  the  same  time,  contract  obligations  for  ATD  and  SD&D  have  declined  significantly 
more  steeply  than  overall  DoD  R&D.  As  discussed  in  the  Federal  R&D  Contracting  in 
Context  section,  this  does  not  appear  to  be  the  result  of  “thoughtful  prioritization  of 
resources;”  rather,  it  appears  that  the  disparate  levels  of  cuts  across  the  DoD’s  R&D 
contracting  portfolio  are  primarily  the  result  of  late-stage  development  programs  for  major 
weapons  systems  either  maturing  out  of  development  or  being  cancelled,  with  a  dearth  of 
new  major  development  programs  starting  in  recent  years. 

Within  the  Army,  R&D  contract  obligations  declined  notably  more  steeply  than  in  the 
DoD  overall  since  2012.  Army  contract  obligations  for  Basic  Research  and  ACD&P  have 
declined  notably  more  slowly  than  overall  Army  R&D  under  sequestration  and  its  aftermath. 
SD&D  was  nearly  steady  over  that  same  period,  but  that  is  a  factor  of  the  near-complete 
disappearance  of  SD&D  contract  obligations  prior  to  2012  due  to  the  cancellation  of  the  FCS 
program  and  the  Army’s  inability  to  start  and  sustain  new  development  programs  for  major 
weapons  systems  in  recent  years.  Meanwhile,  contract  obligations  for  Applied  Research, 
ATD,  and  Operational  Systems  Development  all  declined  significantly  more  steeply  than 
overall  Army  R&D. 

Although  the  distribution  of  cuts  is  different  within  the  Navy’s  R&D  contracting 
portfolio  since  2012,  the  degree  to  which  the  cuts  are  unevenly  distributed  is  similar  to  the 
Army.  Between  2012  and  2014,  Navy  contract  obligations  for  SD&D  and  Operational 
Systems  Development  declined  roughly  in  parallel  to  overall  Navy  R&D.  Obligations  for 
Applied  Research  and  ACD&P  were  relatively  preserved,  with  Applied  Research,  in 
particular,  declining  at  less  than  one-third  the  rate  of  overall  Navy  R&D.  By  contrast, 
obligations  for  Basic  Research  and  ATD  declined  more  steeply  than  overall  Navy  R&D. 

Within  the  Air  Force,  contract  obligations  for  Basic  Research  and  Applied  Research 
both  declined  at  less  than  half  the  rate  of  overall  Air  Force  R&D  since  2012,  while 
Operational  Systems  Development  actually  increased  significantly.  Meanwhile,  obligations 
for  ATD,  ACD&P,  and  SD&D  all  declined  notably  more  steeply  than  overall  Air  Force  R&D, 
with  SD&D  declining  at  nearly  double  the  rate. 

Within  the  MDA’s  R&D  contracting  portfolio,  ACD&P  declined  at  double  the  rate  over 
overall  MDA  R&D  since  2012,  while  Basic  Research  fell  by  over  five  times  the  overall  MDA 
R&D  rate  of  decline.  During  the  same  2012-2014  period,  both  Applied  Research  and  ATD 
saw  notable  increases. 

NASA 

NASA’s  R&D  contracting  portfolio  saw  a  mild  decline  between  2012  and  2014,  but 
much  like  the  DoD  and  its  major  components,  the  cuts  to  NASA  do  not  appear  to  have  been 
done  on  a  “salami  slice”  basis.  Obligations  for  Basic  Research  and  SD&D  saw  notable 
increases,  while  obligations  for  Applied  Research,  ATD,  and  ACD&P  declined  at  rates 
between  two  and  four  times  as  steep  as  for  overall  NASA  R&D. 
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HHS 

In  the  wake  of  sequestration  and  its  aftermath,  HHS  R&D  contract  obligations  were 
virtually  stable  between  2012  and  2014.  But  this  stability  masks  wildly  disparate  increases 
and  decreases  between  the  categories  of  R&D  that  make  up  significant  portions  of  the  HHS 
R&D  contracting  portfolio.  Obligations  for  Basic  Research  and  ATD  both  increased 
significantly  from  2012-2014,  with  the  latter  increasing  by  more  than  a  third.  Obligations  for 
Applied  Research  declined  steeply  over  the  same  period,  while  obligations  for  GOCO  saw  a 
moderate  decline. 

Initial  Findings 

The  data  provides  no  evidence  to  support  the  hypothesis  the  cuts  in  the  wake  of 
sequestration  and  its  aftermath  would  be  done  on  a  “salami  slice”  basis;  in  fact,  the  data 
shows  wildly  divergent  trends  between  different  stages  of  R&D. 

Hypothesis  2:  Newer  R&D  Contracts  Will  Bear  a  Disproportionate  Share  of  Cuts 
During  Budget  Drawdowns. 

The  basis  of  this  hypothesis  is  the  idea  that  established,  ongoing  R&D  programs 
develop  constituencies  and  stakeholders,  both  inside  and  outside  of  government,  that  have 
an  interest  in  seeing  the  program  continue  and  succeed.  As  such,  when  cuts  have  to  be 
made  during  a  budget  drawdown,  it  makes  sense  that  those  constituencies  and 
stakeholders  would  try  to  protect  those  established  programs.  CSIS  thus  theorized  that  in  a 
time  of  budgetary  downturn,  newer  R&D  contracts  would  bear  a  disproportionate  share  of 
the  declines  in  R&D  contract  obligations.  If  this  hypothesis  were  true,  CSIS  would  expect 
that,  within  the  major  R&D  contracting  agencies  and  their  major  components,  the  share  of 
R&D  contract  dollars  obligated  under  “new”  contracts  in  each  fiscal  year  would  decline 
during  the  current  budget  drawdown.  CSIS  refers  to  these  “new”  contracts  in  each  year  as 
“new  start”  contracts. 

Figure  6  shows  the  share  of  contract  dollars  in  each  fiscal  year  that  was  obligated 
under  contracts  originating  in  that  fiscal  year  for  each  of  the  major  R&D  contracting 
agencies.7 


7  Because  FY2000  is  the  first  year  in  the  FPDS  dataset  that  CSIS  uses,  it  is  excluded  from  this 
analysis,  as  all  contract  obligations  in  that  year  are  shown  as  originating  in  FY2000,  even  if  they  come 
from  a  contract  that  began  earlier. 
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Figure  6.  Share  of  R&D  Contract  Obligations  Under  “New  Start”  Contracts,  by 

Customer,  2001-2014 

(FPDS;  CSIS  analysis) 

Department  of  Defense 

The  overall  DoD  R&D  contracting  portfolio  does  not  show  a  consistent  trend  of 
reduced  obligations  for  “new  start”  contracts  during  the  current  budget  drawdown.  The  share 
of  DoD  R&D  contract  obligations  in  each  year  awarded  under  “new  start”  contracts  declined 
from  55%  in  2001  to  a  low  of  21%  in  2007.  The  share  began  to  increase  in  subsequent 
years,  and  that  increase  continued  through  the  early  years  of  the  budget  drawdown,  rising  to 
32%  by  2010.  Over  the  next  three  years,  that  share  fluctuated  between  28%  and  32%, 
peaking  at  34%  in  2014. 

Figure  7  shows  the  share  of  R&D  contract  obligations  awarded  under  “new  start” 
contracts  for  each  of  the  major  DoD  R&D  contracting  components. 
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Figure  7.  Share  of  Defense  R&D  Contract  Obligations  Under  “New  Start” 

Contracts,  by  Component,  2001-2014 

(FPDS;  CSIS  analysis) 

The  share  of  Army  R&D  contract  obligations  under  “new  start”  contracts  declined, 
albeit  not  consistently,  in  the  years  prior  to  the  current  budget  drawdown,  falling  from  48%  in 
2001  to  24%  in  2006.  The  share  obligated  under  “new  start”  contracts  rose  in  subsequent 
years  and  continued  to  rise  during  the  current  budget  drawdown,  reaching  a  high  of  41%  in 
2013  before  falling  back  to  38%  in  2014. 

For  the  Navy,  “new  start”  contract  obligations  accounted  for  over  60%  of  total  R&D 
contract  obligations  in  2001  and  2002,  but  that  share  declined  precipitously  in  2003,  to  29%. 
The  share  declined  gradually  over  the  next  several  years  to  15%  in  2009,  but  rose  to  21%  in 
2010.  Between  2010  and  2014,  the  share  of  Navy  R&D  contract  obligations  under  “new 
start”  contracts  remained  between  20%  and  22%. 

Within  the  Air  Force’s  R&D  contracting  portfolio,  the  share  obligated  under  “new 
start”  contracts  fell  from  50%  in  2001  to  24%  in  2007.  The  share  increased  over  the  next  few 
years  to  37%  by  2010,  fell  to  29%  in  201 1 ,  and  increased  to  45%  by  2014. 

For  the  MDA,  after  the  anomalous  2001 ,  the  share  of  contract  obligations  under  “new 
start”  contracts  fluctuated  below  20%  until  2009,  when  the  share  rose  to  25%.  After  a  drop 
to  20%  in  2010,  the  share  of  MDA  R&D  contract  obligations  under  “new  start”  contracts  rose 
to  37%  by  2012  before  dropping  back  below  20%  in  2013  and  2014. 

NASA 

NASA’s  share  of  R&D  contract  obligations  under  “new  start”  contracts  was  highly 
volatile  in  the  early-to-mid-2000s,  but  since  2008,  that  share  has  remained  between  6%  and 
12%  each  year,  with  no  discernable  pattern  (aside  from  relative  stability)  during  the  budget 
drawdown. 

HHS 

HHS  R&D  contract  obligations  under  “new  start”  contracts  have  been  highly  volatile 
throughout  the  2001-2014  period,  likely  a  function  of  the  smaller  obligation  totals  involved. 
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Since  2008,  the  “new  start”  share  has  fluctuated  between  32%  and  41%  in  all  but  one  year 
(20%  in  2012);  like  NASA,  aside  from  that  relative  stability,  there  is  no  discernable  pattern 
present. 

Department  of  Energy 

The  DoE  data  in  Figure  6  shows  the  degree  to  which  the  DoE  R&D  contracting 
portfolio  is  dominated  by  long-running  contracts.  Between  2008  and  2015,  “new  start” 
contracts  never  exceeded  0.7%  of  total  DoE  R&D  contract  obligations  in  any  year  and 
accounted  for  0.02%  or  less  in  four  of  the  last  five  years. 

Initial  Findings 

The  data  provides  no  support  for  the  hypothesis  that  “new  start”  R&D  contract 
obligations  were  disproportionally  affected  during  the  budget  drawdown;  rather,  in  each 
case,  “new  start”  R&D  contract  obligations  either  were  stable,  increased,  or  else  showed  no 
discernable  trend  over  the  period. 

Hypothesis  3:  Large  Prime  Vendors  Will  Account  for  Increasing  Shares  of  Federal 
R&D  During  Budget  Drawdowns. 

This  hypothesis  can  be  considered  a  companion  to  Hypothesis  3,  because  they 
could  both  be  effects  of  a  similar  cause.  Because  large,  high-profile,  mid-to-late  stage  R&D 
programs  are  the  most  likely  to  have  developed  constituencies  and  stakeholders  that  would 
fight  to  protect  them  during  a  budget  drawdown,  Hypothesis  2  theorized  that  those  R&D 
contracts  would  be  relatively  protected.  And  since  those  large,  high  profile  R&D  programs 
are  likely  to  be  performed  by  large,  high-profile  prime  vendors,  Hypothesis  3  theorizes  that 
R&D  contract  obligations  to  those  same  large,  high-profile  prime  vendors  would  be  relatively 
preserved. 

Department  of  Defense 

Figure  8  shows  DoD  R&D  contract  obligations  to  prime  vendors,  from  2000-2014, 
broken  down  by  the  share  going  to  the  different  vendor  size  categories.8 


8  CSIS  classifies  vendors  into  four  size  categories:  “Small”  vendors  follow  the  government’s 
classification  for  small  businesses,  with  a  couple  of  adjustments  implemented  by  the  study  team; 
“Large”  vendors  are  any  vendors  with  over  $3  billion  in  annual  revenue  from  all  sources;  and 
“Medium”  vendors  are  any  vendors  that  are  neither  small  nor  large.  The  fourth  category,  the  “Big  5” 
vendors  (Lockheed  Martin,  Boeing,  Northrop  Grumman,  Raytheon,  and  General  Dynamics),  are 
separated  out  from  “Large”  due  to  the  outsized  role  they  play  in  federal  contracting  overall. 
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Figure  8.  Defense  R&D  Contract  Obligations  by  Size  of  Vendor,  2000-2014 

(FPDS;  CSIS  analysis) 

Contrary  to  Hypothesis  3,  the  DoD  has  actually  seen  a  dramatic  decline  in  the  share 
of  contract  obligations  going  to  large  prime  vendors.  While  the  “Large”  category  has  held 
steady  through  the  drawdown  and  throughout  most  of  the  2000-2014  period,  the  share  of 
R&D  contract  obligations  going  to  the  Big  5  vendors  has  fallen  from  57%  in  2009  to  42%  in 
2014.  This  is  primarily  the  result  of  the  previously  discussed  five-year  trough  in  the  DoD’s 
development  pipeline  for  major  weapons  systems:  In  recent  years,  as  many  large 
development  programs  were  either  cancelled  or  matured  into  production,  the  DoD  has  been 
largely  unable  to  start  and  sustain  new  large-scale  development  programs.  And  because 
those  large-scale  development  programs  for  major  weapons  systems  are  predominantly 
performed  by  the  Big  5  vendors,  those  vendors  have  borne  the  brunt  of  the  decline  in  DoD 
R&D  contract  obligations. 

Unsurprisingly,  this  trend  is  present  to  an  even  greater  degree  within  Army  R&D. 
While  the  share  of  Army  R&D  contract  obligations  awarded  to  large  vendors  has  remained 
relatively  steady  in  recent  years  (aside  from  a  brief  spike  in  2012  and  2013),  the  share 
awarded  to  the  Big  5  vendors  has  fallen  from  48%  in  2009  to  just  20%  in  2014.  Due  to  the 
Army’s  particularly  severe  issues  with  starting  and  sustaining  new  development  programs  in 
recent  years,  this  trend  is  unlikely  to  reverse  in  the  near  future. 

The  Navy  and  Air  Force  have  seen  declines  in  the  share  of  R&D  contract  obligations 
to  the  Big  5  vendors  more  in  line  with  the  trend  for  DoD  R&D  overall.  Within  the  Navy’s  R&D 
contracting  portfolio,  the  share  of  R&D  contract  obligations  going  to  the  Big  5  vendors  fell 
from  65%  in  2009  to  37%  in  2014.  For  the  Air  Force,  the  share  going  to  the  Big  5  vendors 
fell  from  47%  in  2009  to  31%  in  2014.  In  both  cases,  the  share  awarded  to  large  vendors 
has  been  relatively  stable  over  the  period. 

NASA 

Figure  9  shows  NASA  R&D  contract  obligations  from  2000-2014,  broken  down  by 
the  share  going  to  the  different  vendor  size  categories. 
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Figure  9.  NASA  R&D  Contract  Obligations  by  Size  of  Vendor,  2000-2014 

(FPDS;  CSIS  analysis) 

Unlike  the  DoD  and  its  major  components,  NASA  actually  does  show  increasing 
shares  of  R&D  contract  obligations  going  to  large  prime  vendors.  This  increase,  however, 
began  before  the  current  budget  drawdown;  the  Big  5  vendors  accounted  for  only  26%  of 
NASA  contract  obligations  in  2007,  but  that  share  rose  to  48%  by  2013  before  a  slight 
decline  in  2014.  This  increase  in  Big  5  share  is  primarily  concentrated  in  Basic  Research 
(from  11%  in  2007  to  47%  in  2013)  and  SD&D  (from  7%  in  2007  to  78%  in  2014.)  Thus,  it 
appears  that  the  rising  share  of  NASA  R&D  contract  obligations  going  to  large  prime 
vendors  is  not  attributable  to  factors  relating  to  the  budget  drawdown. 

HHS 

The  Big  5  vendors  have  never  accounted  for  more  than  1%  of  HHS  R&D  contract 
obligations.  The  HHS  has  seen  an  increase  in  the  share  of  R&D  contract  obligations 
awarded  to  large  vendors,  but  this  is  a  trend  that  started  prior  to  the  current  budget 
drawdown.  The  primary  factor  in  this  increase  is  the  increase  in  contract  obligations  for 
GOCO  in  2008  and  2009,  of  which  over  three-quarters  were  awarded  to  large  vendors. 

None  of  the  other  major  categories  within  the  HHS  R&D  contracting  portfolio  have  seen 
consistent  and  notable  increases  or  decreases  in  the  share  of  obligations  awarded  to  large 
prime  vendors  during  the  current  drawdown. 

Initial  Findings 

The  data  provides  no  support  for  the  hypothesis  that  large  prime  vendors  would  see 
increasing  shares  of  R&D  contract  obligations  during  the  current  budget  drawdown.  In  fact, 
in  most  cases,  the  largest  vendors  have  seen  their  shares  decline  precipitously. 

Hypothesis  4:  During  Budget  Drawdowns,  R&D  Will  Be  Increasingly  Funded  Out  of 
Non-R&D-Focused  Funding  Accounts. 

The  theory  of  Hypothesis  4  is  that,  as  budgets  decline,  agencies  may  look  to  fund 
R&D  out  of  budget/funding  accounts  that  are  not  traditionally  R&D-focused  in  order  to  make 
up  for  funding  shortfalls  in  the  R&D-focused  accounts  and  preserve  funding  levels  for  high- 
priority  R&D  programs.  If  this  hypothesis  were  accurate,  the  study  team  would  expect  to  see 
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increases  in  the  share  of  R&D  contracting  obligations  funded  out  of  particular  funding 
accounts  that  were  not  traditionally  the  primary  funding  sources  for  R&D  contracts  within  the 
agency. 

A  couple  of  methodological  notes  related  to  this  analysis: 

•  The  fields  that  allow  for  cross-walking  between  contract  obligations  and 
budget  data  only  began  to  be  filled  in  reliably  in  FY201 1  for  non-DoD 
agencies  and  FY2012  for  the  DoD. 

•  CSIS  focuses  on  funding  accounts,  rather  than  the  higher-level  budget 
accounts,  because  of  the  increased  data  granularity  and  also  because  there 
is  no  consistent  budget  account  schema  between  agencies. 

Department  of  Defense 

Unsurprisingly,  nearly  three-quarters  of  DoD  R&D  contract  obligations  are  funded  out 
of  the  various  DoD  Research,  Development,  Test,  and  Evaluation  (RDT&E)  accounts,  with 
the  Air  Force  and  Defense-wide  accounts  accounting  for  the  largest  shares.  Since  2012,  the 
share  of  DoD  R&D  contract  obligations  funded  out  of  the  Defense-wide  RDT&E  account  has 
risen  from  21%  to  28%,  and  the  share  funded  out  of  the  Air  Force  RDT&E  account 
increased  from  21%  to  23%.  Meanwhile,  the  share  funded  out  of  the  Navy’s  RDT&E  account 
fell  from  18%  to  15%,  while  the  share  funded  out  of  the  Army’s  RDT&E  account  fell  from  7% 
to  6%. 

For  the  other  DoD  funding  accounts  with  non-trivial  levels  of  R&D  contract 
obligations,  there  was  a  mix  of  increases  and  decreases,  though  most  were  relatively  stable. 
The  share  of  R&D  contract  obligations  funded  out  of  the  Navy’s  Aircraft  Procurement 
account  doubled  from  2%  to  4%,  while  the  share  funded  out  of  the  Air  Force’s  Missile 
Procurement  account  fell  from  4%  to  1%.  Additionally,  the  share  funded  out  of  the  Army’s 
Operations  and  Maintenance  (O&M)  account  fell  from  5%  to  3%. 


HHS 


For  the  most  part,  the  shares  of  HHS  R&D  contract  obligations  funded  out  of 
particular  HHS  funding  accounts  have  been  relatively  consistent  from  2011-2014,  with  a 
couple  of  exceptions.  The  share  funded  out  of  the  National  Institute  of  Health’s  (NIH) 
National  Institute  of  Drug  Abuse  rose  from  1%  in  201 1  to  4%  in  2014,  while  the  share 
funded  out  of  the  main  NIH  account  fell  from  25%  in  2012  to  between  19%  and  20%  in  2013 
and  2014. 


NASA 


Unlike  the  other  two  agencies,  there  have  been  significant  shifts  in  the  distribution  of 
R&D  contract  obligations  within  NASA’s  major  funding  accounts.  The  share  of  R&D  contract 
obligations  funded  out  of  the  Cross  Agency  Support  account  rose  from  11%  in  201 1  to  22% 
in  2014,  and  the  share  funded  out  of  the  general  Science  account  rose  from  17%  to  28%. 
Meanwhile,  the  share  funded  out  of  the  Exploration  account  fell  from  27%  to  20%,  the  share 
funded  out  of  the  Human  Space  Flight  account  fell  from  1 1  %  to  7%,  and  the  share  funded 
out  of  the  Science  (Aeronautics  and  Exploration)  account  fell  from  19%  to  0%. 

Initial  Findings 

Though  the  data  is  mixed,  there  is  no  consistent  trend  that  supports  the  hypothesis 
that  R&D  contract  obligations  are  being  increasingly  funded  out  of  non-traditional  accounts 
during  the  current  budget  drawdown. 
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Final  Thoughts  and  Next  Steps 

The  data  highlighted  in  this  report  clearly  shows  that,  while  federal  R&D  contract 
obligations  have  declined  dramatically  overall,  that  decline  has  not  been  consistent  across 
the  major  R&D  contracting  agencies  and  their  major  components,  or  across  the  different 
stages  of  R&D.  Moreover,  with  very  narrow  exceptions,  none  of  the  predictions  made  by  the 
study  team  regarding  the  effect  of  the  downturn  on  federal  R&D  contracting,  based  on  a 
review  of  the  literature  and  consultations  with  experts,  have  been  borne  out. 

These  initial  results  should  give  analysts  pause,  as  the  data  indicates  that  the 
conventional  wisdom  regarding  how  a  budget  drawdown  would  affect  agencies’  R&D 
contracting  portfolios  is  flawed.  While  CSIS  found  no  evidence  that  the  cuts  to  R&D  reflect  a 
thoughtful,  top-to-bottom  prioritization  of  resources,  there  is  also  no  indication  that  the  cuts 
were  done  on  a  “salami-slice”  basis,  or  that  newer  or  earlier-stage  projects  were  sacrificed  to 
protect  later-stage  programs  with  more  entrenched  stakeholders.  As  a  result,  the  concerns 
about  “seed  corn”  R&D  being  disproportionally  affected  by  the  budget  drawdown  also 
appear  to  be  unfounded,  and  the  predicted  rise  of  market  share  for  large  prime  vendors  has 
not  only  not  occurred,  but  has  developed  strongly  in  the  opposite  direction. 

For  the  DoD,  the  key  finding  from  this  data  is  the  existence  of  a  five-year  trough  the 
development  pipeline  for  major  weapons  systems.  The  Air  Force  looks  likely  to  buck  that 
trend  in  the  coming  years  as  spending  for  the  Long  Range  Strike  Bomber  program  ramps 
up.  However,  the  Navy’s  continued  pushing  back  of  development  timelines  for  programs  like 
the  Ohio-replacement  ballistic  submarine  due  to  budget  constraints,  and  the  Army’s 
continued  uncertainty  about  future  missions,  requirements,  and  resources,  indicate  that  the 
overall  trough  is  likely  to  continue  into  the  foreseeable  future. 

In  the  next  stages  of  this  project,  the  study  team  will  test  additional  hypotheses  and 
disaggregate  federal  agency  and  military  department  figures  in  a  manner  appropriate  to 
each  hypothesis,  for  example,  breaking  out  annual  or  quarterly  results,  looking  at  individual 
major  projects  and  facilities,  or  studying  funding  accounts.  The  study  team  plans  to  expand 
the  analysis  of  federal  R&D  contracting  trends,  both  in  terms  of  contract  characteristics  and 
the  supporting  vendor  base,  as  well  as  looking  at  trends  in  R&D-related  grants  to  provide 
additional  context.  CSIS  will  also  continue  to  work  to  identify  and  interview  relevant  experts 
to  help  understand  the  causes  and  effects  of  the  trends  identified  within  the  data. 

Disclaimer 

The  Center  for  Strategic  and  International  Studies  (CSIS)  does  not  take  specific 
policy  positions;  accordingly,  all  views  expressed  in  this  presentation  should  be  understood 
to  be  solely  those  of  the  author(s). 
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Identifying  and  Mitigating  the  Impact  of  the  Budget 
Control  Act  on  High  Risk  Sectors  and  Tiers  of  the  Defense 
Industrial  Base:  Assessment  Approach  to  Industrial  Base 

Risks 

Lirio  Aviles — is  the  Fixed  Wing  and  Shipbuilding  Assessment  Lead  in  the  Office  of  the  Deputy 
Assistant  Secretary  of  Defense  for  Manufacturing  and  Industrial  Base  Policy  (ODASD[MIBP]).  In  this 
role,  Aviles  advises  the  DASD  on  all  matters  relating  to  the  defense  industrial  base,  including 
industrial  capabilities  assessments,  acquisitions  and  consolidation,  preservation  of  critical  industries 
and  technologies,  and  opportunities  for  international  cooperation.  From  2012  to  2013,  Aviles  worked 
in  the  Joint  Strike  Fighter  Program  Office,  where  she  served  as  a  Prognostics  and  Health 
Management  Engineer.  Prior  to  that,  she  participated  in  the  Air  Force  Education  with  Industry 
Program.  During  that  time,  she  worked  on  the  development,  integration,  and  test  of  a  real-time 
hyperspectral  image  system  for  the  Joint  Improvised  Explosive  Devices  Defeat  Organization.  From 
2008  to  201 1 ,  she  was  a  Test  &  Evaluation  (T&E)  Advisor  for  the  Rotary  Wing  Branch  at  the  Air  Force 
Life  Cycle  Management  Center.  Previously,  she  worked  as  a  Human  Factors  Engineer  in  the  Air 
Force  Test  Center  at  Edwards  AFB.  She  started  her  career  as  a  Project  Staff  Engineer  with  Hanes 
Menswear,  where  she  was  in  charge  of  transforming  the  manufacturing  operations  from  traditional 
production  lines  to  work  teams.  Aviles  earned  a  bachelor’s  degree  in  industrial  engineering  from  the 
University  of  Puerto  Rico.  She  has  an  MBA  with  concentrations  in  industrial  management  and  human 
resources  from  the  Interamerican  University,  [lirio.n.aviles.civ@mail.mil] 

Sally  Sleeper — is  the  Director  of  the  Strategy,  Doctrine,  and  Resources  Program  at  the  RAND 
Corporation.  She  joined  the  Office  of  the  Deputy  Assistant  Secretary  of  Defense  for  Manufacturing 
and  Industrial  Base  Policy  (ODASD[MIBP])  as  the  senior  advisor  in  July  2012.  In  this  role,  she  was 
responsible  for  developing  tools  and  techniques  to  allow  analysis  of  the  sub-tier  supplier  network; 
developing  policy  to  mitigate  risk  to  National  Security;  and  assessing  DoD  programs,  budgets, 
strategies,  investments,  and  business  combinations  that  affect  defense  industry  related  material 
production  and  supply.  Dr.  Sleeper  comes  to  the  Department  of  Defense  from  the  RAND  Corporation, 
where  she  was  a  Senior  Management  Scientist.  From  2008  to  2012,  she  was  the  Director  of 
Programs  for  the  RAND  Gulf  States  Policy  Institute,  with  offices  in  New  Orleans,  LA,  and  Jackson, 

MS.  In  that  role,  she  worked  to  develop  projects  in  Louisiana,  Mississippi,  and  Alabama  in  areas 
where  analysis  can  help  inform  decision  making  and  make  a  difference  in  the  region,  such  as  coastal 
protection  and  restoration,  healthcare,  and  workforce  development.  Her  research  areas  include 
innovation  and  technology  development,  regional  economic  development,  and  organizational 
effectiveness.  Previously,  Dr.  Sleeper  was  a  private-sector  Transportation  Management  Analyst.  She 
received  a  bachelor’s  degree  in  environmental  design  and  planning  from  the  University  of  Colorado  at 
Boulder,  an  MS  in  policy  analysis  and  public  management  from  Stony  Brook  University,  and  an  MS  in 
organization  theory  and  a  PhD  in  organization  science  and  economics  from  Carnegie  Mellon 
University,  [sleeper@rand.org] 

Abstract 

The  Department  of  Defense  (DoD)  requires  insight  into  the  risks  that  the  Budget  Control  Act 
(BCA)  is  placing  on  the  defense  industrial  base  (DIB),  particularly  in  those  sectors  that  have 
been  previously  identified  as  critical  and  at  high-risk  of  losing  critical  capabilities.  The  Office 
of  the  Deputy  Assistant  Secretary  of  Defense  for  Manufacturing  and  Industrial  Base  Policy 
(ODASD[MIBP])  has  developed  a  methodology  to  identify  the  impact  of  budget  cuts  on  the 
DIB.  During  2014  and  2015,  the  MIBP  identified  capabilities  provided  by  the  DIB  that  were  at 
high  risk  of  being  compromised  or  unavailable  to  the  warfighter  using  the  fragility  and 
criticality  methodology  and  implemented  mitigation  plans  to  sustain  the  industrial  base. 
Funding  to  execute  mitigation  plans  was  included  in  the  FY16  Presidential  Budget.  The  MIBP 
created  an  assessment  approach  to  evaluate  the  impact  of  the  BCA  on  the  DIB.  Only  the 
sectors  and  tiers  previously  identified  as  high  risk  were  assessed.  The  framework  evaluates 
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the  loss  of  design  and  manufacturing  skills,  loss  of  innovation,  loss  of  competition,  and  loss  of 
infrastructure.  In  addition,  potential  DoD  steps  to  sustain  high  risk  sectors,  sub-sectors,  and 
tiers  under  a  BCA  environment  were  identified.  DoD  leadership  is  using  the  results  to  inform 
resource  decision  making. 

Introduction 

The  industrial  base  is  an  integral  part  of  the  Department  of  Defense  (DoD)  force 
structure  needed  to  provide  the  highest  performance  and  innovative  capabilities  to  the 
warfighter.  However,  the  current  budgetary  situation  is  forcing  industry  to  make  business 
decisions  that  will  have  long  term  consequences  in  the  nation’s  ability  to  advance  its 
technological  capabilities.  Defense  industry  consolidations,  challenges  incentivizing  new 
entrants  to  the  DoD’s  critical  markets,  and  loss  of  design  teams  and  manufacturing  skills  due 
to  procurement  reductions  are  some  of  the  main  factors  threatening  the  industrial  base. 
Consolidation  trends  have  led  to  the  creation  of  six  “mega-prime”  providers  today — reducing 
competition  and  creating  barriers  to  entry  due  to  their  sheer  financial  size.  Budget 
uncertainty  and  industry’s  perception  of  DoD  contracting  practices  and  intellectual  property 
protection  limit  the  interest  of  non-traditional  companies  from  working  with  the  DoD. 
Procurement  and  research  and  development  (R&D)  programs,  which  have  been  delayed  or 
cancelled,  also  have  an  impact  on  industry’s  ability  to  retain  its  design  teams  and  exercise 
the  critical  manufacturing  skills  for  defense-unique  products. 

While  budget  swings  are  not  new  to  the  DoD,  the  trends  and  challenges  discussed 
above  are  impacting  today’s  defense  industrial  base  (DIB)  and  limiting  its  ability  to  support 
the  technological  superiority  requirements  of  the  Department.  In  addition,  appropriations 
have  consistently  fallen  short  of  carefully  planned  President’s  Budgets. 

As  illustrated  in  Figure  1 ,  the  DoD  and  industry  will  have  to  overcome  several 
budgetary  challenges: 


•  The  Services  need  to  balance  force  structure,  readiness,  and  capability  to 
meet  national  security  commitments  in  their  President’s  Budget  submissions. 
Programs  like  the  Ohio  Replacement  and  Long  Range  Strike  Bombers  are 
part  of  the  U.S.  strategy  to  modernize  nuclear  weapons  systems  and  the 
number  one  priority  for  the  Navy  and  AF,  respectively.  In  order  to  fund  these 
programs,  the  Services  will  have  to  make  other  procurement,  readiness,  or 
force  structure  trade-offs.  These  decisions  are  extremely  difficult  due  to 
competing  priorities  and  their  effect  on  the  long-term  strategies. 

•  Current  programs  are  moving  from  design  and  manufacturing  stages  to 
operations  and  maintenance.  This  situation  creates  a  design  and 
manufacturing  gap  that  puts  at  risk  the  industry’s  ability  to  sustain  and 
exercise  the  critical  skills  for  the  advanced  weapons  systems  required  in  the 
future. 

•  As  the  war  winds  down  and  U.S.  forces  reduce  their  role  in  active  combat,  the 
declining  demand  for  some  defense-unique  products  adds  pressure  to  mid 
and  lower  tiers  of  the  industrial  base  that  depend  on  DoD  business  to  achieve 
their  minimum  sustainment  rates. 

•  Budgetary  uncertainty  has  contributed  to  companies’  adoption  of  an  income- 
focused  strategy  as  defense  firms  invest  in  share  buy-backs,  dividends,  and 
mergers  or  divestitures  to  create  income  and  improve  profitability.  Without  the 
ability  to  plan  for  future  programs,  industry  is  reluctant  to  invest  in  R&D,  yet 
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the  DoD  relies  on  industry’s  R&D  for  innovation,  technical  dominance,  and 
increased  efficiency. 


DoD  Procurement  Funding  PB2011-PB2016 


Prior  Actuals 


PB2014 

PBZOli 


-  PBZ016 

M  PB2013 


PB2015 

PB2012 


Figure  1.  DoD  Investments  on  Procurement — Actuals  vs.  Presidential  Budget 

(PB) 

The  current  situation  of  the  defense  industrial  base  is  exacerbated  by  the  Budget 
Control  Act  (BCA)  of  2011,  which  proposed  DoD  spending  reductions  of  approximately  10% 
annually  for  the  next  10  years.  Figure  2  provides  a  summary  of  the  events  related  to  the 
BCA  and  the  effect  on  the  FY16  PB  for  the  DoD.  In  the  National  Defense  Authorization  Act 
of  FY15,  Congress  expressed  concerns  about  the  effect  of  the  BCA  on  the  industrial  base. 
Consequently,  Congress  requested  the  Office  of  the  Secretary  of  Defense  to  provide  an 
analysis  of  sectors  and  tiers  of  the  private  industrial  base  found  to  be  at  highest  risk,  and 
how  the  risk  assessment  has  changed  since  enactment  of  the  BCA  of  201 1 .  This  paper 
outlines  the  framework  developed  by  the  Manufacturing  and  Industrial  Base  Policy  (MIBP) 
Office  to  assess  the  industrial  base  risks  and  provides  a  summary  of  the  results. 
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Reached  Federal  Debt  Ceiling 

*  Outstanding  Debt  Approaching  Debt-Ceiling  Limit  (516.4T) 


FY1 6  PS  Request  for  DOD 
$534B 


Budget  Control  Act  of  2011  (S.  365)  Enacted  BCA  Cap  for  FY1 6 

-  S900B  in  savings  over  1 Q  years;  -194.7  billion  in  Defense  budget  cuts  $498B 


Sequestration  (2013) 

-  Automatic  spending  cuts  went  into  effect  in  2013 

V _ 

j 

r 

Bipartisan  Budget  Act  (2013) 

BBA13  Cap  forFYlS 

*  Raises  the  sequestration  caps  for  fiscal  years  2014  and  2015 

*  Extend  caps  until  2023;  bill  is  projected  to  lower  the  deficit  by  $23  billion  S498B 


Bipartisan  Budget  Act  (2015)  CaP  f0F FY^ 

*  Raises  the  sequestration  caps  for  fiscal  years  2016  and  2017 

*  Extend  caps  until  2029  $522B 

*  Additional  S-8E  that  can  be  used  for  oco  or  as  part  of  the  base  budget 

V _ _ _ _ _ _ J 


Figure  2.  Budget  Control  Act  Events  and  Effect  on  FY16  Presidential  Budget  (PB) 

for  the  DoD 


Defining  Industrial  Base  Risks 

The  DoD  defines  industrial  base  risks  as  uncertainties  regarding  industry’s  ability  to 
design,  manufacture,  and  sustain  the  DoD’s  present  and  future  critical  capabilities.  A  critical 
capability  is  defined  as  a  capability  difficult  to  replace  if  disrupted.  A  critical  capability  will 
have  a  combination  of  the  following  characteristics:  defense-unique;  requires  specialized 
skills  to  integrate,  manufacture,  or  maintain  the  capability;  requires  defense-specific 
knowledge  to  reproduce  this  capability,  an  alternative,  or  the  next  generation  design; 
requires  the  use  of  specialized  equipment  and/or  facilities  for  manufacturing  and 
sustainment;  the  time  required  to  restore  the  capability  will  have  a  negative  impact  on  the 
mission;  and  the  availability  of  alternatives  to  meet  DoD  needs  without  the  capability.  The 
Office  of  the  Deputy  Assistant  Secretary  of  Defense  for  Manufacturing  and  Industrial  Base 
Policy  (ODASD[MIBP])  uses  FaC1  assessments  to  identify  critical  capabilities. 

The  MIBP  developed  risk  definitions  (see  Tables  1  and  2)  to  assess  the  industrial 
base  risks  for  each  sector,  sub-sector,  tier,  and  sub-tier,  as  required.  In  some  cases,  the 
evaluation  considered  a  sub-sector,  while  in  other  cases,  a  tier  or  sub-tier  was  assessed.  A 
sector  refers  to  the  big  segments  of  the  industrial  base  providing  similar  or  related  products 
and  services  in  a  given  market.  A  sub-sector  divides  the  sectors  based  on  more  specific 
activities  and/or  products.  Tiers  define  the  specific  components  and  services  required  to 
manufacture  a  final  product.  Sub-tiers  divide  the  components  and  services  into  specific 


1  In  201 1 ,  the  MIBP  was  tasked  with  developing  a  forward-leaning  approach  that  could  identify  the 
cumulative  effect  on  vital  capabilities  of  procurement  decisions  across  programs  and  Services.  The 
organization  used  the  existing  1 996  framework  to  develop  a  methodology  that  could  be  used 
proactively,  across  Services  and  industrial  sectors,  that  is  rigorous,  repeatable,  and  transparent.  That 
methodology  to  assess  the  industrial  base  is  known  as  the  Fragility  and  Criticality  assessment 
process,  or  FaC  for  short. 
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products,  similar  to  those  found  in  a  bill  of  material.  Figure  3  provides  an  example  of  the 
relationship  between  sectors,  sub-sectors,  and  tiers. 


Figure  3.  Aircraft — Sector,  Sub-Sectors,  and  Tiers 

The  analysis  framework  was  based  on  the  two  risk  components:  likelihood  and 
consequence.  The  risk  level  ranges  from  low  to  high  based  on  the  likelihood  of  losing  a 
critical  capability  and  the  ability  to  reconstitute  the  capability  once  it  is  lost. 
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Table  1.  Industrial  Base  Risk  Definition — Likelihood 

Risk 

Level 

Definition 

Low 

Low  expectation  that  a  critical  capability  will  be  lost,  and,  if  lost,  the  capability  is 
easily  reconstituted. 

*  Industry  can  quickly  respond  to  DoD  requirements  on  time  and  at 
reasonable  cost 

*  Competition  exists;  design  and  manufacturing  skills  are  being  exercised 
through  multiple  programs;  R&D  programs  are  funded 

*  Adequate  infrastructure  is  available  to  meet  requirements 

Medium 

Medium  expectation  that  a  critical  capability  may  be  lost  and,  if  lost,  will  not  easily 
be  reconstituted. 

*  Industry  has  few  qualified  companies  fora  capability;  declining 
procurement  may  reduce  domestic  competition 

*  Mitigation  plans  are  in  place  to  address  a  capability  where 
reconstitution  would  be  very  costly  or  to  maintain  domestic  source  of 

supply 

■  High  expectation  that  a  critical  capability  will  be  lost,  and,  if  lost,  will  be  difficult  or 
impossible  to  reconstitute. 

*  Industry  has  one  source  of  supply,  and  declining  procurement  increases 
the  chances  of  exit 

*  No  mitigation  action  in  place;  reconstitution  would  significantly  increase 
cost  and  schedule 

*  Industry  will  have  difficulty  responding  to  DoD  requirements  on  time 
and/or  at  a  reasonable  cost 

Table  2  describes  the  consequences  of  losing  a  critical  capability.  Consequences  are 
defined  according  to  five  main  areas  that  are  critical  to  design,  develop,  test,  and  sustain 
current  and  future  weapons  systems:  design  skills,  manufacturing  skills,  innovation, 
competition,  and  infrastructure.  One  risk  may  have  consequences  in  multiple  areas. 
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Table  2.  Industrial  Base  Risks  Definitions — Consequences 


Industrial  Base  Risks 

Consequences 

Definition 

Loss  of  Design  Skills 

Loss  of  defense-specific  knowledge  required  to  reproduce  a  critical  capacity, 
an  alternative,  or  the  next  generation  design 

Loss  of  Manufacturing 

Skills 

Loss  of  specialized  skills  needed  to  integrate,  produce,  or  sustain  a  critical 
capability 

Loss  of  Innovation 

Reductions  in  RDT&E  funding  that  will  jeopardize  technology-based  programs. 
Industry  is  focusing  R&D  investments  on  near-term  payoffs. 

Loss  of  Competition 

Procurement  levels  that  cannot  sustain  multiple  suppliers  will  lead  companies 
to  exit  the  market.  Markets  also  may  consolidate  through  increased  mergers 
and  acquisitions  and  partnerships  between  primes  and  suppliers. 

Loss  of  Infrastructure 

Loss  of  specialized  equipment  or  facilities  needed  to  integrate,  manufacture, 
or  maintain  a  critical  capability.  Lack  of  investment  to  maintain  and 
modernize  the  equipment,  tooling,  and  facilities  needed  to  sustain  the 
capability. 

Defining  Industrial  Base  Sectors  at  Risk 

The  MIBP  used  the  results  of  FaC  assessments  conducted  between  2013  and  2014, 
industrial  base  reports,  and  inputs  from  subject  matter  experts  to  identify  sectors  at  high  risk 
of  losing  critical  capabilities,  considering  factors  like  current  and  future  demand,  acquisition 
phase  of  major  programs,  and  mitigation  strategies.  The  following  sectors  were  identified  at 
higher  risk: 

•  Missiles  and  Munitions  Sector — The  missile  and  munitions  sector  is 
comprised  of  the  DoD’s  smart  bombs  and  tactical  and  strategic  missiles.  This 
sector  is  primarily  a  defense-unique  industrial  sector  and,  therefore,  is  highly 
dependent  on  the  DoD’s  demand.  Over  the  past  decade,  the  munitions  and 
missile  sector  has  provided  no  new-start  missile  opportunities,  as  all  “new” 
missile  programs  have  been  designated  as,  or  have  become,  upgrades  to 
existing  systems. 

•  Space  Sector — The  space  sector  is  primarily  driven  by  the  commercial 
market  and  includes  satellites,  launch  services,  ground  systems,  networks, 
payloads,  propulsion,  and  electronics.  Although  the  commercial  focus  of  this 
sector  allows  leveraging  the  commercial  technology  advancement,  security 
restrictions  limit  the  benefits.  Therefore,  the  DoD  must  remain  vigilant  in  order 
to  maintain  critical  capabilities  that  are  specialized  for  military  applications 
and  have  very  low  demand  compared  with  commercial  products. 

•  Aircraft  Sector — The  aircraft  sector  is  comprised  of  commercial  and  defense 
aircraft.  Defense  aircraft  are  divided  in  three  main  sub-sectors:  fixed-wing, 
rotary  wing,  and  unmanned  systems.  The  fixed-wing  sub-sector  includes 
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fighters,  bombers,  cargo,  and  transportation  aircraft.  The  rotary  wing  sub¬ 
sector  includes  helicopters  used  for  combat,  combat  support,  and  services. 
Unmanned  aircraft  systems  include  the  necessary  components,  network,  and 
personnel  to  control  an  unmanned  aircraft,  including  a  launching  element,  if 
needed.  There  has  been  a  steady  decline  in  the  number  of  defense 
development  programs  for  fixed-wing  and  rotary  wing  aircraft.  Modernization 
programs  will  help  sustain  important  capabilities,  but  will  not  provide 
opportunities  for  major  design,  development,  and  integration  work.  Design 
shortfalls  are  also  projected  because  much  of  the  defense  aerospace 
workforce  is  close  to  retirement,  and  the  pool  of  young  engineers  available  to 
replace  them  is  migrating  to  other  industries. 

•  Shipbuilding  Sector — The  defense  shipbuilding  sector  is  comprised  of 
seven  shipyards  and  other  shipyards  which  concentrate  on  commercial  ships, 
but  will  periodically  enter  and  exit  the  naval  market.  The  U.S.  shipbuilding 
industrial  base  depends  on  DoD  business  to  sustain  critical  design  and 
manufacturing  skills,  as  well  as  to  maintain  their  current  infrastructure. 

•  Combat  Vehicles  Sector — The  ground  vehicle  sector  is  generally 
categorized  in  two  broad  vehicle  classes:  tactical  wheeled  vehicles  (TWV) 
and  combat  vehicles.  The  TWV  are  usually  commercial  trucks  modified  for 
military  use  in  demanding  environments  and/or  missions.  This  type  of  truck 
benefits  from  dual-use  or  commercial  demand.  Combat  vehicles,  on  the  other 
hand,  are  typically  heavily  armored  and  integrated  with  complex  weapons 
and  systems;  therefore,  they  have  limited  commercial  application.  This  sector 
faces  a  number  of  industrial  base  challenges,  including  retaining  critical 
design  and  integration  skills,  as  well  as  sustaining  critical  suppliers  in  the  sub¬ 
sector  tiers. 


Specific  sub-sectors,  tiers,  and  sub-tiers  were  identified  in  each  of  the  sectors 
previously  mentioned.  Information  about  the  specific  risk  is  not  discussed  in  this  paper  to 
protect  business  sensitive  and  pre-decisional  information  used  in  the  analysis.  However,  an 
example  of  the  aircraft  sector,  which  has  been  openly  discussed  by  DoD  leadership,  is 
provided  in  the  next  sections. 

Risk  Level  Assessment 

Risk  level  assessments  for  each  of  the  sectors,  sub-sectors,  and  tiers  identified  at 
high  risk  were  conducted  using  the  following  timeframes: 

•  FY1 1  (baseline)  -  BCA  enactment 

•  FY1 3  -  Bipartisan  Budget  Act  enactment 

•  FY1 5  -  Current  FY  at  the  time  of  the  assessment 

•  FY16  -  Most  current  guidance  for  investment  on  the  next  five  years  at  the 
time  of  the  analysis 

The  assessment  was  based  on  the  number  of  DoD  programs  supporting  the  sectors 
at  high  risk  over  the  time  periods  under  evaluation  and  the  acquisition  phase  of  those 
programs. 

The  final  product  was  a  risk  level  matrix  for  each  of  the  industrial  areas.  Figure  4 
provides  an  example  of  the  aircraft  sector  assessment.  In  this  case,  the  assessment  was 
done  at  the  sub-sector  level,  fixed-wing-fighter  aircraft.  In  201 1 ,  there  were  multiple 
programs  in  manufacturing,  and  the  F-35  program  was  supporting  development  activities.  In 
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2013,  there  were  still  multiple  programs  in  production,  but  the  F-22  closed  their  production 
line  and  the  F-35  development  activities  decreased.  By  2015,  most  of  the  fighter  programs 
were  transitioning  from  a  manufacturing  phase  to  operations  and  sustainment.  In  addition, 
no  new  design  work  for  fighters  was  expected,  creating  a  development  gap  until  the  2020s, 
when  new  fighter  programs  are  expected  to  start.  The  FY16  PB  included  funds  to  start  a 
program  known  as  the  Aerospace  Innovation  Initiative  (All).  This  program  will  help  to  sustain 
the  development  skills  required  to  produce  the  next-generation  of  fighters,  maintain 
competition  in  the  sector,  and  promote  innovation. 

Although  the  medium  level  indicates  that  mitigation  plans  are  in  place  to  address  the 
risk,  capabilities  in  the  medium  risk  level  will  be  highly  dependent  on  budget  decisions. 
Funding  for  mitigation  plans  may  be  transferred  or  delayed  in  order  to  fund  higher  priorities 
within  the  Department.  Sub-sectors  in  this  risk  level  should  be  monitored  constantly. 


Sector:  Aircraft 

2011 

2013 

2015 

PB2016 

Sub-Sector:  Fixed  Wing  -  Fighters 

• 

• 

Figure  4.  Industrial  Base  Risk  Level  (Likelihood) — Aircraft  Example 

Identifying  the  Effect  of  the  BCA  on  the  Defense  Industrial  Base 

Funding  cuts  due  to  the  BCA  will  create  additional  barriers  to  overcome  the  current 
challenges.  However,  the  impact  of  the  BCA  cannot  be  assessed  in  isolation.  Decreasing 
procurement  and  R&D  funds,  Services’  priorities,  the  scheduled  end  of  multiple  DoD 
programs,  and  corporate  strategic  plans  are  other  factors  that  will  impact  the  industrial  base 
and  are  considered  when  making  decisions  related  to  the  BCA. 

The  MIBP  used  the  following  sources  to  determine  the  impact  of  the  BCA: 

•  Presidential  Budget — PB16  projects  funding  levels  for  FY16  to  FY20.  The 
trends  in  procurement  and  R&D  budgets  provide  a  good  indication  of  the 
expected  investments  in  the  defense  aircraft  sector  (see  Figures  5  and  6). 
The  fighter  procurement  and  RDT&E  funding  stay  relatively  steady  from  2018 
to  2020  due  to  the  F-35  program.  However,  there  is  no  new  fighter 
development  or  procurement  during  that  period  of  time.  The  decreasing  trend 
in  procurement  and  RDT&E  investment  may  be  worsened  by  the 
implementation  of  BCA  cuts. 
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Fixed- Wing  Procurement  and  RDT&E  Funding 
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Figure  5.  DoD  Investments  in  Fixed-Wing  Procurement  and  RDT&E  (PB16) 


Fighter  Sub-  Sector  Procurement  and  RDT&E  Funding 
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Figure  6.  DoD  Investments  in  Fighter  Aircraft  Procurement  and  RDT&E  (PB16) 

•  Subject  matter  experts  (SMEs)  from  the  Services  and  representatives  of 
multiple  DoD  offices  evaluated  the  potential  impact  of  BCA  enactment  in 
FY16  to  their  current  programs  and  plans.  The  following  potential  impacts 
were  identified  for  the  fixed-wing-fighter  sub-sector: 
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o  Aerospace  Innovation  Initiative  (All)  funds  may  be  eliminated. 

o  RDT&E  programs  to  advance  sixth  generation  fighter  technology  may 
be  reduced  or  eliminated. 

o  BCA1 6-driven  divestiture  or  reduction  of  aircraft  fleets  may  affect 
primes  and  lower  tier  suppliers  that  are  essential  to  capabilities 
sustainment. 

•  SMEs  applied  the  definitions  in  Table  2,  industrial  Base  Consequences,  to 
assess  expected  consequences  if  a  capability  is  lost  due  to  a  BCA  cut 
implementation. 


Sector:  Aircraft 

Loss  of  Design 
Skills 

Loss  of 

Manufacturing 

Skills 

Loss  of 
[□□.ovation 

Loss  of 
Competition 

Loss  of 
Infrastruclnnc 

Sub-Sector:  Fixed  Wing  -  Fighters 

•J 

-J 

7 

Figure  7.  Industrial  Base  Risks  (Consequences) 

Updated  Risk  Assessment — The  risk  level  was  updated  to  reflect  the  potential  impact 
of  the  BCA  in  FY16.  It  is  important  to  note  that  the  actual  BCA  impacts  will  depend  on  the 
Services’  budget  and  decision  priorities  at  the  time  of  the  BCA  cuts  implementation.  In  the 
case  of  the  fixed-wing-fighters  sub-sector,  the  likelihood  of  losing  critical  capabilities 
increased. 


Sector:  Aircraft 


Sub-Sector:  Fixed  Wing  -  Fighters 
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Figure  8.  Industrial  Base  Risk  Assessment  Including  Potential  BCA  Impact 
The  Final  Outcome 

To  finalize  the  analysis,  a  combination  of  all  the  factors  assessed  was  provided  for 
each  sector  and  its  respective  sub-sectors  and  tiers  (see  Figure  9).  DoD  leadership  will  use 
this  combination  of  factors  to  determine  the  industrial  base  risk  levels  and  consequences  in 
specific  areas  of  the  industrial  base  and  make  decisions  about  the  potential  cuts.  For 
example,  in  Figure  8,  the  fighter  aircraft  risk  level  is  expected  to  change  from  medium  to 
high  if  BCA  cuts  are  implemented.  This  could  represent  the  elimination  or  delay  of  funds  for 
a  new  program  or  implementation  of  a  mitigation  strategy.  BCA  implementation  will  require 
reducing  or  re-programming  funds  based  on  priorities.  Leadership  will  use  these  data  to 
establish  priorities  based  on  the  risk  level  and  consequences  they  are  willing  to  accept.  The 
risk  level  needs  to  be  paired  with  the  consequence  when  establishing  priorities. 
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ImJuvIruil  Base  Risks  at  RCA  Levels  (( 'em  sequence) 
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example,  the  Adaptive  Engine 
Transfer  Program  (AETP)  is 
working  on  engine  technology  for 
sixth-generation  fighters. 
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Figure  9.  Potential  Impact  of  BCA  on  Industrial  Base — Fixed-Wing  Fighters 

Example 


Conclusion 

The  results  of  the  analysis  provided  the  following  conclusions: 

•  BCA  levels  would  have  a  significant  negative  impact  on  major  sectors  of  the 
defense  industrial  base:  The  FY16  Presidential  Budget  included 
considerations  and  mitigation  strategies  necessary  for  a  healthy  industrial 
base  capable  of  providing  critical  capabilities  to  the  DoD.  However,  many  of 
the  DoD’s  remediation  efforts  to  protect  high  risk  sectors  and  tiers  may  be  at 
risk  under  the  BCA. 

•  The  DoD’s  future  actions  to  reduce  the  potential  impact  of  BCA16  on  the 
industrial  base  will  depend  on  the  cuts  across  the  Services  to  reduce  costs 
while  balancing  force  structure,  readiness,  and  capability  to  meet  current  and 
future  national  security  demands. 

•  Policy  changes  and  additional  actions  may  be  necessary  to  sustain  the 
industrial  base. 
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•  The  DoD  can  take  the  following  steps  to  help  sustain  high-risk  sectors  and 
tiers  under  a  BCA  environment: 

o  Develop  acquisition  strategies  that  promote  competition  while 
sustaining  design  and  manufacturing  skills. 

o  Expand  the  use  of  available  tools  and  program2  to  mitigate  industrial 
base  risks. 

o  Continue  working  on  FaC  assessment  to  identify  critical  capabilities  at 
risk  and  develop  mitigation  strategies  through  groups  like  the  Joint 
Industrial  Base  Working  Group  (JIBWG)3  and  the  Industrial  Base 
Counsel  (IBC).4 
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2  Examples  of  industrial  base  tools  and  programs  include  the  Industrial  Base  Analysis  and 
Sustainment  (IBAS)  funds,  the  ManTech  program,  and  Title  III.  IBAS  provides  temporary  sustainment 
for  critical  defense-related  industrial  capabilities  that  are  temporarily  at  risk  of  being  lost.  ManTech 
provides  the  primary  investment  mechanism  for  enabling  defense  essential  manufacturing  capability. 
Title  III  authorizes  economic  incentives  to  create,  expand,  or  preserve  critical  domestic  industrial 
manufacturing  resources. 

3  The  JIBWG  conducts  industrial  base  assessments  and  recommends  investment  priorities. 

4  The  purpose  of  the  IBC  is  to  drive  a  forward-looking  view  of  the  defense  industrial  base  enterprise, 
ensure  alignment  to  overarching  objectives,  and  enable  more  effective  decision  making  at  all  levels. 
This  group  is  comprised  of  executive  leadership  who  will  set  priorities  for  the  defense  industrial  base, 
including  assessments,  risk  mitigation,  and  a  clear  pathway  for  escalation  of  issues. 
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